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JET  ENGINE  MECHANIC-AFSC  426X2:  EXPERIMENTAL 
JOB  PERFORMANCE  TESTS 


INTRODUCTION 


Historically,  the  Air  Force  has  used  technical  school  grades  (TSGs) 
as  a criterion  for  validating  aptitude  measures.  However,  TSGs  have 
not  proven  to  be  a completely  acceptable  criterion.  One  of  the  primary 
reasons  for  the  unacceptability  of  TSGs  is  the  weak  relationship  between 
school  grades  and  later  job  performance. 

The  purpose  of  this  study  was  to  develop  a job-related  criterion 
metric  against  which  aptitude  tests  could  be  validated.  In  line  with 
this  objective,  it  was  desired  that  the  new  criterion  metric  be  more 
indicative  of  on-the-job  performance  than  are  TSGs.  A constraint,  im- 
posed by  the  high  developmental  and  administrative  costs  of  job  sample 
tests,  was  that  the  new  criterion  metric  be  a paper-and-pencil  measure 
of  job  performance. 

The  use  of  paper-and-pencil  measures  usually  entails  lower  fidelity 
in  simulating  job  tasks  than  do  actual  job  performance  items.  This 
decrement  usually  results  from  the  fact  that  paper-and-pencil  tests  do 
not  cover  a sufficient  number  of  job-relevant  tasks.  An  approach  to 
the  problem  of  task  coverage  is  to  concentrate  upon  comprehensively 
identifying  job  components  through  task  analysis.  A second  problem 
with  paper-and-pencil  items  is  that  they  often  do  not  cover  the  complete 
range  of  skills  within  a particular  task  category.  To  insure  fidelity, 
it  is  necessary  that  all  task  categories  be  adequately  sampled,  and 
that  the  complete  range  of  skills  within  categories  be  covered.  Each 
item  on  the  paper-and-pencil  measure  then  maps  to  a specific  task  cate- 
gory and  skill  level  within  the  task  category. 

Once  the  relevant  job  skills  have  been  identified  and  suitable 
paper-and-pencil  measures  developed,  a link  must  still  be  established 
between  job  knowledge,  as  assessed  on  the  criterion  measure,  and  actual 
job  performance.  The  current  study  makes  the  assumption  that  career 
progression  is  a direct  measure  of  job  performance.  That  is,  the  selec- 
tion processes  of  the  Air  Force  tend  to  promote  job  incumbents  who  suc- 
cessfully meet  their  job  requirements.  Similarly,  when  an  incumbent 
ceases  to  satisfactorily  perform  his  duties,  promotion  stops  or  slows 
down. 

Following  the  assumption  that  career  progression  reflects  job 
performance,  the  current  study  makes  the  further  assumption  that,  to 
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the  extent  test  performance  is  associated  with  career  level,  the 
Criterion  Tests  (CTs)  may  be  considered  as  indices  of  actual  job 
proficiency. 

Employing  an  indirect  measure  of  job  proficiency  such  as  career 
progression  does,  however,  entails  a certain  amount  of  risk.  A pri- 
mary element  of  risk  is  that  there  may  be  less  relationship  between 
job  proficiency  and  career  progression  than  initially  assumed.  The 
factors  which  account  for  career  progression  may  only  indirectly 
involve  job  knowledge.  The  Criterion  Test  items  may,  indeed,  reflect 
job  knowledge,  but  fail  to  discriminate  skill  groups.  However,  if 
Criterion  Tests  differentiate  various  skill  groups,  they  then  may  be 
said  to  at  least  be  a measure  of  "job  success,"  with  job  success  being 
defined  as  long-term  job  incumbency. 

Given  the  foregoing  objectives  and  assumptions,  the  Criterion 
Test  development  effort  proceeds  as  follows.  First,  the  Air  Force 
Occupational  Survey  (AFOS)  is  used  to  identify  the  content  of  Air 
Force  Specialty  Code  (AFSC)  426X2,  Jet  Engine  Mechanic  (formerly 
432X0).  The  task  responsibilities  of  3,  5,  and  7 skill  level  person- 
nel in  this  AFSC  are  separately  identified.  Finally,  a large  pool  of 
potential  paper-and-pencil  test  items  (70  percent  more  than  planned) 
are  designed  in  such  a way  as  to  discriminate  1 and  3,  3 and  5,  and 
5 and  7 skill  level  job  incumbents.  The  ability  of  the  CTs  to  dis- 
criminate career  progression  levels  is  the  link  between  test  perform- 
ance and  job  proficiency.  An  initial  editing  and  screening  of  the  item 
pool  to  remove  defective  items  results  in  tryout  versions  of  the  Cri- 
terion Tests. 

The  next  step  in  the  process  is  to  administer  the  CTs  to  a large 
group  of  subjects  representative  of  each  skill  level  in  the  target 
AFSC.  Results  from  this  administration  are  used  to  select  the  "best" 

CTs.  Best,  in  this  context,  refers  to  CTs  made  up  of  those  items 
having  the  highest  discriminating  power,  the  most  homogeneity,  and 
the  most  acceptable  distribution  of  responses  to  the  distractors. 

After  defective  items  have  been  removed,  and  the  final  versions 
of  the  CTs  assembled,  the  study  proceeds  with  an  investigation  of  test 
discriminability  and  reliability.  As  previously  discussed,  the  ability 
of  the  Criterion  Test  scores  to  discriminate  career  levels  links  the 
CT  results  with  job  proficiency.  It  is  also  desirable  to  demonstrate 
that  the  responses  of  subjects  to  the  CT  items  are  consistent. 

The  last  phase  of  the  study  involves  the  relationships  of  the 
Criterion  Tests  and  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB) , the  standard  Air  Force  aptitude  measure  used  in  the  selection 
and  classification  of  enlisted  personnel.  ASVAB- Criterion  Test  relation- 
ships are  of  interest  since  selection  validity  hinges  upon  being  able 
to  predict  test  performance,  and,  thus,  indirectly,  job  proficiency, 
from  ASVAB  scores.  The  relationships  between  TSGs  and  the  ASVAB  scales 
permit  a comparison  of  the  present  results  with  previous  validity 
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studies.  ASVAB-TSG  associations,  in  comparison  with  ASVAB-Criterlon 
Test  correlations,  also  give  some  Indication  of  how  changing  the  cri- 
terion for  selection  will  possibly  affect  the  composition  of  the  pool 
of  jet  engine  mechanics.  Finally,  should  Criterion  Test  results  be 
associated  with  career  progression,  the  relationship  of  TSGs  with  the 
Criterion  Tests  will  give  an  indication  of  how  technical  school  per- 
formance relates  to  eventual  job  proficiency.  Each  of  these  issues 
is  discussed  in  further  detail  in  the  next  two  sections.  Research 
Methodology  covers  the  development  and  evaluation  of  individual  test 
items.  The  discriminability  and  reliability  of  the  tests  is  addressed 
in  the  Results  section.  The  Results  section  and  the  Validity  Studies 
section  address  the  relationships  among  the  Criterion  Tests,  the  ASVAB, 
and  technical  school  grades. 


RESEARCH  METHODOLOGY 


Task  Analysis  and  Item 
Pool  Development 

The  first  step  in  developing  a pool  of  pa per- and- pencil  job  perform- 
ance items  was  to  systematically  identify  the  separate  task  components 
of  AFSC  426X2  using  the  AFOS.  Various  job  components  were  then  grouped 
into  broad  categories  of  activities  called  task  domains.  The  task  domains 
identified  for  each  skill  level  are  presented  in  Table  1. 


TABLE  1 TASK  DOMAINS  OF  ASFC  426X2 


Level-3 


1.  Safety 

2.  Tools  and  Equipment 

3.  Inspections 

4.  Technical  Orders,  Forms,  Maintenance  Levels 

5.  Security 

6.  Shipping,  Storage,  Preservation 

7.  Jet  Engine  Theory 

8.  Jet  Engine  Operation  and  Maintenance 


Level- 5 

1.  Safety 

2.  Tools  and  Equipment 

3.  Inspections 

4.  Technical  Orders,  Forms,  Maintenance  Levels 

5.  Security 

6.  Shipping,  Storage,  Preservation 

7.  Jet  Engine  Theory 

8.  Jet  Engine  Operation  and  Maintenance 

9.  Training 

10.  Work  Performance 

11.  Non-Work  Personnel  Issues 

12.  Supply 
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TABLE  1 (continued) 


Level- 7 

1.  Safety 

2.  Tiols  and  Equipment 

3.  Inspections 

4.  Technical  Orders,  Forms,  Maintenance  Levels 

5.  Security 

6.  Shipping,  Storage,  Preservation 

7.  Jet  Engine  Theory 

8.  Jet  Engine  Operation  and  Maintenance 

9.  Work  Performance 

10.  Non-Work  Personnel  Issues 

11.  Supply 

12.  Evaluation 

13.  Planning 

14.  Supervision 

15.  Training 

1 

I 

: As  may  be  noted,  the  task  domains  for  each  of  the  three  skill 

levels  have  some,  but  not  complete,  overlap.  New  domain  elements  are 

; added  as  the  skill  level  of  job  incumbents  increases.  For  example, 

"Training"  is  a domain  element  for  level  5,  but  not  level  3.  The  fact 
that  a particular  domain  element  is  on  more  than  one  list  does  not  mean, 
however,  that  test  items  representing  that  element  are  the  same  across 
different  skill  levels.  Clearly,  the  activities  involved  in  a job  ele- 
ment like  "Safety"  are  different  for  a level  3 mechanic  and  a level  7 
noncommissioned  officer  (NCO) . 

Following  identification  of  the  relevant  task  domains,  job  charts, 
or  systematic  matrices  of  skill  components  which  make  up  the  job  of  jet 
engine  mechanic,  were  prepared.  The  job  matrices  were  compiled  separately 
for  each  skill  level.  Specifically,  the  job  matrices  identified: 

1.  The  tasks  done  by  most  job  incumbents  at  that  skill 
level. 

2.  The  tools  and  equipment  used. 

3.  The  knowledge,  skills  and  judgments  needed  to 
perform  the  task. 

4.  Tasks  generally  not  performed  by  mechanics  at 
the  next  lower  skill  level,  or,  if  performed  at 
the  lower  level,  done  in  a way  requiring  different 
skills,  knowledge,  or  judgments. 
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Using  the  information  contained  in  the  job  analysis  matrices, 
a large  pool  of  potential  test  items  was  then  developed.  Guide- 
lines for  item  development  were  as  follows: 

1.  Items  should  measure  job  performance  on  tasks 
done  by  most  mechanics  at  a particular  skill 
level  (e.g.,  3,  5,  and  7). 

2.  Items  should  discriminate  between  the  perform- 
ance of  mechanics  at  one  skill  level  and  the 
performance  of  those  at  the  next  lowest  skill 
level,  e.g.,  1 versus  3,  3 versus  5,  and 

5 versus  7. 

The  unrefined  item  pool  was  next  submitted  to  the  Air  Force 
for  final  approval.  Test  items  were  then  edited  and  screened  to 
eliminate  obvious  defectives.  The  pre-screening  process  resulted 
in  a pool  of  450  items  (150  in  each  test  for  inclusion  in  the  try- 
out versions  of  the  Criterion  Tests). 

It  should  be  noted  at  this  point  that  the  job  performance  Cri- 
terion Tests  developed  in  the  current  study  do  not  have  the  same 
objective  as  the  Specialty  Knowledge  Tests  (SKTs)  used  in  the  Weighted 
Airman  Promotion  System  (WAPS).  The  SKTs  are  developed  on  the  basis 
of  training  documentation  and  represent  training  standards  of  performance. 
On  the  other  hand,  the  Criterion  Tests  were  designed  to  reflect 
on-the-job  performance — that  is,  job  performance  as  it  actually 
occurs  in  the  field. 


Data  Collection 


Following  Air  Force  approval  of  the  final  sets  of  test  items, 
and  the  preparation  and  printing  of  the  testing  materials,  data 
collection  began  at  12  Air  Force  bases.  The  preliminary  data  collec- 
tion plan  is  shown  in  Table  2.  More  subjects  than  actually  required 
were  scheduled  at  each  base.  Over- scheduling  was  done  to  serve  as 
a precaution  against  collecting  unusable  data,  or  failing  to  obtain 
the  expected  number  of  subjects  at  some  bases.  The  numbers  planned 
for  each  base  were  a function  of  the  actual  number  of  each  skill 
level  reported  to  be  available.  Testing  on  each  group  of  subjects 
was  scheduled  to  be  completed  during  one  working  day.  To  control 
for  possible  fatigue  and  order  effects,  the  order  of  administration 
of  the  various  tests  was  counterbalanced. 
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TABLE  2 PLANNED  FREQUENCIES  OF  SUBJECTS  WITHIN 
PAFSC  426X2  ACROSS  SKILL  LEVELS  AT 
SELECTED  AIR  FORCE  BASES 


Skill  Level 


Air  Force  Base 

3 

5 

2 

Totals 

Sejmiour  Johnson 

30 

25 

35 

90 

McGuire 

25 

40 

40 

105 

Dover 

35 

85 

42 

162 

Little  Rock 

38 

20 

32 

90 

Barksdale 

28 

18 

26 

72 

Columbus 

28 

5 

28 

61 

Nellis 

25 

35 

12 

72 

Travis 

80 

80 

40 

200 

George 

26 

9 

37 

72 

Castle 

28 

70 

64 

162 

Mountain  Home 

28 

28 

34 

90 

Holloman 

48 

15 

45 

108 

Totals: 

419 

1284 

As  expected,  the  actual  number  of  subjects  obtained  was  lower  than 
the  number  sought.  The  actual  numbers  of  subjects  obtained  in  each 
skill  level  at  each  Air  Force  base  are  presented  in  Table  3. 


TABLE  3 OBTAINED  FREQUENCIES  OF  SUBJECTS  WITHIN 
PAFSC  426X2  ACROSS  SKILL  LEVELS  AT 
SELECTED  AIR  FORCE  BASES 


Skill  Level 


Air  Force  Base 

3 

5 

2 

Totals 

Seymour  Johnson 

0 

26 

9 

35 

McGuire 

21 

20 

7 

48 

Dover 

55 

72 

50 

177 

Little  Rock 

29 

35 

38 

102 

Barksdale 

22 

6 

21 

49 

Columbus 

9 

1 

17 

27 

Nellis 

6 

31 

18 

55 

Travis 

4 

46 

19 

69 

George 

5 

15 

15 

35 

Castle 

22 

34 

30 

86 

Mountain  Home 

9 

30 

29 

68 

Holloman 

12 

20 

30 

62 

Totals: 

m 

336 

283 

813 

7 


During  approximately  the  same  time  period  that  job  performance 
data  was  being  collected  from  active  duty  jet  engine  mechanics,  the 
Air  Force  was  also  administering  each  Criterion  Test  and  the  ASVAB 
to  422  basic  airmen  (level  1 personnel).  Results  from  419  level  1 
subjects  were  eventually  usable.  The  complete  test  data  file  thus 
contains  scores  from  1232  subjects— 813  experienced  mechanics  and 
419  basic  airmen. 
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RESULTS 


Item  Selection 

Following  the  administration  of  the  Criterion  Tests  and  the  ASVAB, 
individual  results  were  assembled  and  prepared  for  analysis.  The  first 
step  in  the  data  analysis  was  to  obtain  each  subject's  score  on  each  of 
the  Criterion  Tests  and  the  nine  ASVAB  scales.  The  Criterion  Test  scores 
were  corrected  for  guessing  using  the  standard  formula  used  with  multiple- 
choice  tests.  The  resulting  means,  standard  deviations,  25th,  and  75th 
percentile  scores  for  each  group  on  the  three  Criterion  Tests  are  given 
in  Table  4. 


TABLE  4 SCORES  ON  THE  THREE  CRITERION 
TESTS  BY  SKILL  LEVEL 


(Total  N = 

1,232;  1 

= 419;  3 = 194; 

5 = 336;  7 = 

283) 

Criterion  Skill 

Standard 

25th 

75th 

Test 

Level 

Mean 

Deviation 

Percentile 

Percent: 

1 

1 

9.41 

8.91 

3 

14 

1 

3 

75.15 

21.86 

63 

90 

1 

5 

72.32 

22.25 

60 

88 

1 

7 

93.57 

16.03 

86 

103 

2 

1 

7.46 

8.38 

0 

12 

2 

3 

21.48 

14.04 

12 

31 

2 

5 

28.13 

15.21 

18 

38 

2 

7 

54.78 

17.48 

43 

66 

3 

1 

3.57 

6.94 

-2 

7 

3 

3 

10.35 

7.86 

4 

15 

3 

5 

16.51 

11.69 

8 

23 

3 

7 

38.00 

14.19 

27 

47 

Referring  to  Table  4,  the  mean  test  scores  on  all  three  Criterion 
Tests  generally  increase  as  a function  of  skill  level.  One  exception 
is  CT  1,  where  level  5 personnel  do  not  score  as  high  on  the  average  as 
level  3 personnel.  The  quartile  scores  for  each  group  indicate  that  the 
distributions  of  test  scores  are  nearly  symmetric  about  individual  group 
means. 
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An  analysis  of  variance  (ANOVA)  performed  on  the  three  test  scores 
across  skill  levels  indicates  that  the  group  means  are  significantly 
different.  Partial  ANOVA  results  are  presented  in  Table  5. 


TABLE  5 PARTIAL  ANOVA  RESULTS  FOR  THE  THREE 
150-ITEM  CRITERION  TESTS 

(Total  N = 1,232) 


Test 

F-Statistic 

Degrees  of  Freedom 

P 

1 

1654.22 

3,  1228 

<.0001 

2 

679.83 

3,  1228 

<.0001 

3 

632.11 

3,  1228 

<.0001 

The  next  step  in  the  test  development  effort  was  to  reduce  the 
number  of  items  in  each  Criterion  Test  by  selecting  the  100  "best"  items 
in  each  test.  "Best,"  in  this  context,  refers  to  those  items  which  most 
closely  meet  the  following  criteria: 

1.  The  item  discriminates  between  the  appropriate 
skill  levels.  For  example,  good  items  in  CT  1 — 
the  criterion  for  skill  level  3 — should  be  answered 
correctly  more  often  by  level  3 personnel  than  by 
basic  airmen  (level  1 personnel). 

2.  An  item  should  be  correlated  with  the  total  score. 

Items  having  a high  correlation  with  the  total 
score  more  consistently  measure  the  job  proficiency 
aspects  on  which  the  whole  test  discriminates  skill 
level  groups.  Job  proficiency  is  considered  to  be 
a unidimensional,  global  construct. 

3.  The  distribution  of  answers  to  the  item  distractors 
should  be  satisfactory.  This  last  check  will  identify 
items  in  which  one  or  more  response  alternatives  are 
rarely  selected  by  the  testees.  Additionally,  tabu- 
lating answer  distributions  will  identify  items  where 
some  Incorrect  response  alternatives  are  marked  more 
often  by  high  scorers  than  by  low  scorers.  Items 
having  these  characteristics  are  defective  and  should 
be  rejected. 

In  applying  the  above  selection  rules,  primary  importance  was  given 
to  the  discriminating  power  of  items.  The  rationale  for  this  choice  is 
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based  upon  the  fact  that  the  overall  purpose  of  the  tests  is  to  serve 
as  criteria  for  their  associated  skill  level.  Thus,  tests  that  do  not 
discriminate  between  career  levels  are  not  suited  for  use  as  criteria. 

If  the  items  do  not  discriminate  career  levels,  then  the  link  between 
the  tests  and  job  proficiency  is  broken.  Selection  rules  2 and  3 were 
included  to  lend  support  to  the  primary  criterion  for  selection.  These 
rules  were  to  be  used  in  the  event  that  item  discrimination  alone  did 
not  provide  sufficient  information  for  deciding  whether  to  retain  or 
reject  an  item. 

Items  on  the  three  Criterion  Tests  were  next  submitted  to  an  item 
analysis  and  the  following  statistics  obtained: 

1.  The  index  of  discrimination  (phi). 

2.  The  rank  of  the  discrimination  index. 

3.  The  item  difficulty  for  the  group  relevant  to  the 
test. 

4.  The  item  variance  for  the  group  relevant  to  the 
test. 

5.  The  item-total  score  correlation  for  the  group 
relevant  to  the  test. 

6.  Ttie  rank  of  the  item-total  score  correlation. 

7.  The  item  difficulty  for  the  skill  group  immediately 
lower  than  the  criterion  group.  This  is  denoted 
the  "non-relevant  'group.'" 

8.  The  item  variance  for  the  non-relevant  group. 

9.  The  item-total  score  correlation  for  the  non- 
relevant  group. 

1 

10.  The  average  rank  of  the  discrimination  index  and  I 

the  item-total  score  correlation  for  the  criterion 
group. 

The  complete  lists  of  item  statistics  for  each  of  the  tests  are  pre- 
sented in  Appendices  1,  2,  and  3.  Items  are  listed  in  order  of  their 
discriminating  power,  as  indicated  by  phi.  Also  included  in  the  last 
column  of  each  appendix  is  a table  entry  indicating  whether  the  item  was 
selected  for  inclusion  in  the  final  versions  of  the  Criterion  Tests. 

A "Yes"  or  a "No"  indicates  this  choice. 

Referring  to  Appendix  1,  which  contains  the  item  analysis  statistics 
for  CT  1,  the  indices  of  discrimination  (phi  coefficients)  computed  across 
items  for  skill  levels  1 and  3 range  from  0.64  to  0.17.  The  corresponding 
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Item  difficulties  Indicate  that  60-90  percent  of  level  3 personnel  gen- 
erally ansver  the  items  correctly.  These  difficulty  levels  indicate  a 
desirable  situation,  since  CT  1 is  designed  to  differentiate  level  3 
from  level  1 personnel.  The  corresponding  item-total  score  correlations 
for  the  retained  items  on  CT  1 range  from  0.49  to  0.14.  None  of  the  re- 
tained items  in  CT  1 has  a negative  correlation  with  the  total  score. 

The  item  analysis  statistics  on  the  second  test  (see  Appendix  2) 
indicate  that  CT  2 does  a relatively  poor  job  of  discriminating  level  5 
from  level  3 personnel.  The  discrimination  coefficients  for  the  two 
relevant  groups  (skill  levels  3 and  5)  range  from  0.21  to  -0.23.  For 
retained  items,  the  range  is  from  0.21  to  .01.  Given  the  sample  size 
involved  (N  = 530),  a value  of  at  least  0.085  is  required  for  a correla- 
tion coefficient  to  be  considered  significantly  different  from  zero  at 
the  p=.05  level  of  significance.  Using  the  p=.05  criterion,  only  31  test 
items  on  CT  2 yield  a discrimination  index  which  is  individually  sig- 
nificantly different  from  zero. 

The  reason  for  the  low  discriminating  power  of  items  in  CT  2 is 
apparent  when  considering  the  item  difficulties  for  the  two  relevant 
groups.  The  problem  is  that  most  items  on  CT  2 are  generally  too  diffi- 
cult for  level  5 personnel.  In  most  cases,  level  5 personnel  answer 
the  items  correctly  more  often  than  level  3 personnel.  However,  the 
magnitude  of  the  difference  is  not  large  enough  to  reliably  differentiate 
the  two  groups.  There  is  too  much  overlap  in  the  response  distributions. 

The  level  of  response  overlap  is  evident  from  the  quartile  scores  pre- 

sented in  Table  4.  Here,  it  is  noted  that  more  than  25  percent  of 
level  3 personnel  score  higher  on  CT  2 than  the  mean  score  of  the 
level  5 personnel. 

CT  3 presents  a situation  midway  between  the  relatively  good  charac- 
teristics of  CT  1 and  the  poor  characteristics  of  CT  2.  On  CT  3,  the 

discrimination  coefficients  for  retained  items  range  from  0.48  to  .01. 
Again,  given  the  sample  size  involved  (N  = 619) , a correlation  of  approxi- 
mately 0.08  is  required  for  significance  at  the  p=.05  level.  Using  that 
criterion,  85  items  on  CT  3 individually  yield  significant  indices  of 
discrimination.  Looking  at  the  corresponding  item  difficulty  statistics 
for  items  with  ((i  <.08,  it  is  again  the  case  that  items  on  CT  3 having 
low  discriminating  power  suffer  from  the  same  problem  as  the  majority 
of  items  on  CT  2:  the  items  are  generally  too  difficult  for  the  cri- 
terion skill  level  personnel. 

In  summary.  Criterion  Tests  2 and  3 do  a less  satisfactory  job 
of  differentiating  their  relevant  skill  groups  than  does  CT  1.  On 
both  CTs  2 and  3,  the  major  problem  concerns  items  that  are  too  diffi- 
cult for  the  criterion  skill  groups.  This  result  may  indicate  that 
differentiating  within  skill  levels  of  jet  engine  mechanics  is  much 
more  difficult  than  simply  differentiating  mechanics  from  non-mechanics. 
One  of  the  difficulties  encountered  in  the  item  development  phase  of 
the  study  was  deciding  specifically  which  task  elements  separated 
level  3 jobs  from  level  5 jobs,  and  level  5 jobs  from  level  7 jobs. 
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There  were  no  clearly  defined  criteria.  In  most  cases,  job  content  dif- 
ferences across  skill  levels  were  a matter  of  the  degree  of  the  skills 
involved.  This  overlap  in  the  job  content  of  the  3,  5,  and  7 level  skill 
groups  is  reflected  in  their  respective  performance  on  the  Criterion  Tests. 

To  recap  the  item  selection  phase  of  the  study.  Tables  6,  7,  and  8 
present  sequential  lists  of  all  retained  Criterion  Test  items  and  the 
corresponding  difficulty  indices  for  all  skill  levels. 
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TABLE  6 ITEMS  RETAINED  IN  CRITERION  TEST  1 WITH 
CORRESPONDING  DIFFICULTY  LEVELS 

(Total  N = 1,232;  1 = 419;  3 = 194;  5 = 336;  7 = 283) 


Skill 

Level 

Pooled 

Skill 

Level  Pooled 

Item 

1 

3 

5 

7 

3-5-7 

Item 

1 

3 

5 

7 ; 

3-5-7 

1 

2 

.24 

.79 

.62 

.74 

.70 

51 

81 

.26 

.93 

.94 

.98  ■ 

.95 

2 

3 

.27 

.77 

.69 

.91 

.79 

52 

82 

.43 

.89 

.88 

.96 

.91 

3 

6 

.15 

.45 

.43 

.77 

.55 

53 

83 

,42 

.81 

.85 

.92 

,86 

4 

8 

.24 

.87 

.82 

.91 

.86 

54 

87 

,27 

.83 

.88 

.97 

.90 

5 

9 

.21 

.85 

.87 

.95 

.90 

55 

89 

.24 

.84 

.80 

.89 

.84 

6 

10 

.19 

.81 

.84 

.89 

.85 

56 

90 

.23 

.86 

.89 

.97 

,91 

7 

14 

.18 

.49 

.54 

.72 

.59 

57 

91 

,44 

,93 

.93 

.99 

.95 

8 

15 

.20 

.71 

.61 

.92 

.74 

58 

92 

.23 

.61 

.62 

.86 

.70 

9 

17 

.27 

.86 

.90 

.87 

.88 

59 

93 

.27 

.59 

.52 

.51 

.53 

10 

18 

.54 

.95 

.92 

.97 

.95 

60 

94 

.30 

.65 

.70 

.89 

.76 

11 

21 

.52 

.91 

.85 

.95 

.90 

61 

95 

.25 

.91 

.90 

.98 

.93 

12 

23 

.14 

.64 

.66 

.88 

.73 

62 

96 

.54 

,86 

.87 

.93 

.89 

13 

26 

.41 

.60 

.61 

.93 

.72 

63 

97 

.29 

.65 

.70 

,87 

.75 

14 

27 

.16 

.54 

.64 

.78 

.66 

64 

98 

.21 

.64 

.54 

.65 

.60 

15 

28 

.40 

.88 

.91 

.87 

.89 

65 

99 

.31 

.89 

.82 

.94 

.88 

16 

31 

.14 

.36 

.43 

.83 

.55 

66 

101 

.19 

.52 

.40 

,44 

.44 

17 

32 

.26 

.80 

.80 

.85 

.82 

67 

102 

.41 

.87 

.79 

.86 

.83 

18 

34 

.33 

.91 

.90 

.98 

.93 

68 

104 

.36 

.79 

.79 

.88 

.82 

19 

35 

.18 

.79 

.80 

.86 

.82 

69 

105 

.22 

.41 

.48 

.80 

.57 

20 

36 

.16 

.74 

.79 

.83 

.79 

70 

106 

.27 

.68 

.63 

.72 

.68 

21 

37 

.18 

.66 

.70 

.92 

.77 

71 

108 

.18 

.54 

.36 

.45 

. 

22 

40 

.62 

.95 

.93 

.96 

.95 

72 

110 

.19 

.47 

.31 

.52 

.42 

23 

41 

.25 

.85 

.87 

.95 

.90 

73 

111 

.26 

.75 

.69 

.79 

.74 

24 

42 

.30 

.81 

.84 

.93 

.87 

74 

112 

.23 

.72 

.68 

,81 

.74 

25 

43 

.38 

.81 

.80 

.88 

.83 

75 

114 

.41 

.60 

.73 

.87 

.75 

26 

44 

.32 

.70 

.64 

.73 

.69 

76 

115 

.62 

.94 

.87 

.98 

.92 

27 

46 

.36 

.75 

.75 

.88 

.80 

77 

116 

.17 

.70 

.68 

.57 

.65 

28 

47 

.11 

.29 

.19 

.82 

.43 

78 

.’7 

.34 

.77 

.84 

.89 

.84 

29 

48 

.37 

.89 

.91 

.95 

.92 

79 

119 

,26 

.54 

.48 

.57 

.53 

30 

49 

.36 

.71 

.68 

.78 

.72 

80 

120 

.23 

.74 

.71 

.92 

.79 

31 

52 

.11 

.61 

.54 

.59 

.58 

81 

121 

.21 

.53 

.33 

.28 

.36 

32 

54 

.41 

.93 

.87 

.97 

.92 

82 

123 

.26 

.70 

.61 

.77 

.69 

33 

55 

.24 

.72 

.75 

.84 

.78 

83 

124 

,23 

.58 

.62 

.83 

.69 

34 

58 

.33 

.74 

.65 

.79 

.72 

84 

126 

.21 

.81 

.80 

.90 

.84 

35 

59 

.39 

.89 

.82 

.90 

.86 

85 

128 

.48 

.80 

.78 

,96 

,85 

36 

61 

.24 

.66 

.77 

.94 

.80 

86 

129 

.11 

.60 

.65 

.85 

.71 

37 

62 

.13 

.71 

. 66 

.78 

.72 

87 

131 

.17 

.63 

.39 

.52 

.‘49 

38 

63 

.59 

.96 

.89 

.98 

.94 

88 

132 

.30 

.75 

.67 

.84 

.75 

39 

64 

.30 

.81 

.83 

.95 

.87 

89 

134 

.22 

.59 

.38 

.22 

.38 

40 

65 

.34 

.76 

.78 

.80 

.78 

90 

135 

,25 

.47 

.43 

.49 

.46 

41 

66 

.42 

.70 

.60 

.56 

.61 

91 

136 

.10 

.46 

.52 

.56 

.52 

42 

68 

.35 

.76 

.78 

.89 

.81 

92 

137 

.18 

.41 

.52 

.83 

.60 

43 

69 

.25 

.79 

.85 

.95 

.87 

93 

138 

.27 

.56 

.62 

.70 

.63 

44 

70 

.38 

.84 

.78 

.75 

.78 

94 

140 

.14 

.38 

,38 

.48 

.42 

45 

71 

.29 

.88 

.87 

.95 

.90 

95 

141 

.17 

.76 

.77 

.82 

.79 

46 

72 

.44 

.90 

.90 

.96 

.92 

96 

142 

.15 

.66 

.58 

.64 

.62 

47 

74 

.31 

.72 

.64 

.69 

.68 

97 

145 

.20 

.55 

.49 

.62 

.55 

48 

75 

.20 

.71 

.62 

.65 

.65 

98 

146 

.11 

.56 

.60 

.81 

.66 

49 

76 

.21 

. 66 

.42 

.42 

.48 

99 

149 

.24 

.48 

.50 

.59 

.53 

50 

77 

.27 

.88 

.89 

.98 

.92 

100 

150 

.33 

.66 

.71 

.73 

.70 

14 


TABLE  7 ITEMS  RETAINED  IN  CRITERION  TEST  2 WITH 
CORRESPONDING  DIFFICULTY  LEVELS 


(Total  N = 1,232;  1 = 419;  3 = 194;  5 = 336;  7 = 283) 


Skill 

Level  i 

Pooled 

Skill 

Level  Pooled 

Item 

1 

3 

5 

7 

3-5-7 

Item 

1 

3 

5 

7 : 

3-5-7 

1 

2 

.13 

.30 

.36 

.48  ■ 

.39 

51 

75 

.19 

.21 

.24 

• 34  ■ 

.27 

2 

3 

.11 

.19 

.31 

.44 

.32 

52 

76 

.43 

,80 

.81 

.95 

.86 

3 

4 

.23 

.28 

.31 

.30 

.30 

53 

77 

.31 

.34 

.36 

.49 

.40 

4 

5 

.25 

.39 

.46 

.66 

.51 

54 

78 

.24 

.30 

.43 

.73 

.50 

5 

7 

.17 

.08 

.12 

.12 

.11 

55 

80 

.36 

.57 

.76 

.94 

.78 

6 

9 

.29 

.53 

.67 

.92 

.72 

56 

82 

.57 

.66 

.71 

.89 

.76 

7 

12 

.35 

.43 

.51 

.58 

.51 

57 

83 

.52 

.59 

.67 

.70 

,66 

8 

13 

.39 

.39 

.44 

.73 

.53 

58 

84 

.15 

.18 

.22 

.36 

,26 

9 

14 

.27 

.22 

.29 

.72 

.42 

59 

85 

.28 

.17 

.20 

.41 

.27 

10 

16 

.20 

.12 

.19 

.65 

.33 

60 

86 

.18 

.58 

.60 

.72 

.64 

11 

18 

.16 

.30 

.37 

.66 

.46 

61 

88 

.47 

,55 

.59 

.74 

.63 

12 

19 

.43 

.57 

.59 

.82 

.67 

62 

89 

.41 

,60 

.63 

.78 

.67 

13 

20 

.29 

.18 

.21 

.40 

.27 

63 

90 

.32 

,55 

.56 

.88 

.67 

14 

22 

.36 

.55 

.67 

.89 

.72 

64 

92 

.29 

.27 

.37 

.27 

.31 

15 

24 

.30 

.27 

.37 

.67 

.45 

65 

93 

.30 

.30 

.35 

.70 

.46 

16 

25 

.38 

.55 

.64 

.76 

.66 

66 

99 

.28 

.40 

.54 

.80 

.60 

17 

26 

.19 

.27 

.29 

.49 

.36 

67 

100 

.26 

.35 

.36 

.66 

.46 

18 

27 

.12 

.11 

.21 

.39 

.25 

68 

101 

.16 

.17 

.18 

.36 

.24 

19 

29 

.32 

.56 

.59 

.59 

.58 

69 

103 

.26 

.35 

.42 

.62 

.47 

20 

30 

.44 

.61 

.62 

.73 

.66 

70 

104 

.27 

.27 

.40 

.80 

.51 

21 

31 

.18 

.61 

.67 

.91 

.74 

71 

105 

.37 

.38 

.42 

.59 

.47 

22 

32 

.23 

.06 

.08 

.08 

.08 

72 

106 

.26 

.30 

.35 

.37 

.34 

23 

33 

.45 

.65 

.72 

.92 

.77 

73 

107 

.25 

.26 

.33 

.26 

.29 

24 

34 

.22 

.30 

.38 

.62 

.44 

74 

108 

.25 

.32 

.35 

.45 

.38 

25 

35 

.18 

. 12 

.14 

.14 

.14 

75 

109 

.33 

.38 

.43 

.62 

,48 

26 

37 

.58 

.76 

.81 

.92 

.84 

76 

111 

.30 

,26 

.30 

.44 

.34 

27 

38 

.15 

.45 

.50 

.67 

.55 

77 

112 

.38 

.45 

.52 

.69 

.56 

28 

39 

.47 

.52 

.62 

.82 

.66 

78 

114 

.41 

,64 

.64 

.71 

.66 

29 

42 

. 31 

. 29 

.33 

.50 

.38 

79 

116 

,31 

.49 

.60 

.70 

.61 

30 

43 

.38 

.63 

.74 

.88 

.76 

80 

117 

.34 

.32 

.35 

.49 

.39 

31 

45 

. 19 

.09 

.13 

,27 

.17 

81 

118 

.24 

.43 

.45 

.52 

.47 

32 

46 

. 11 

. 19 

.23 

.63 

.36 

82 

119 

.23 

.20 

.21 

.16 

.19 

33 

47 

.29 

.20 

,22 

.43 

.29 

83 

121 

.38 

.31 

,40 

.78 

.51 

34 

48 

.29 

.36 

.43 

.66 

.49 

84 

123 

.17 

.25 

.41 

.81 

.51 

35 

50 

.44 

.75 

.82 

.95 

.85 

85 

124 

.33 

.55 

.65 

.75 

.66 

36 

51 

.29 

.46 

.55 

.71 

.58 

86 

127 

.43 

.31 

,36 

.32 

.34 

37 

52 

.20 

. 19 

.22 

.34 

.26 

87 

129 

.28 

.37 

.49 

.58 

.49 

38 

55 

. 20 

.38 

.43 

.56 

.46 

88 

130 

.17 

.24 

.33 

.57 

.39 

39 

57 

. 33 

. 25 

.34 

.73 

.46 

89 

131 

.27 

.20 

.34 

.59 

.40 

40 

59 

. 49 

. 60 

.63 

.71 

.65 

90 

133 

.20 

.17 

.22 

.26 

.22 

41 

60 

.42 

.48 

.61 

.70 

.61 

91 

134 

.27 

.59 

.66 

.76 

.68 

42 

62 

.25 

.49 

.63 

.90 

.69 

92 

135 

.39 

.51 

.56 

.93 

.68 

43 

64 

. 26 

. 30 

.40 

.53 

.42 

93 

136 

.14 

.19 

.19 

.42 

.27 

44 

65 

. 27 

. 25 

.44 

.42 

.39 

94 

138 

.15 

.29 

.48 

.75 

.53 

45 

66 

. 20 

. 15 

.18 

.33 

.23 

95 

139 

.38 

.28 

.32 

.34 

.32 

46 

67 

. 52 

. 69 

.68 

.89 

.76 

96 

142 

.41 

.41 

,48 

.52 

.48 

47 

68 

. 24 

. 16 

.22 

.18 

,19 

97 

144 

.15 

.17 

.21 

.24 

.21 

48 

69 

. 32 

. 42 

.46 

.79 

.56 

98 

145 

.22 

.23 

,35 

.62 

.42 

49 

71 

. 23 

. 24 

.29 

.48 

.35 

99 

147 

.12 

.24 

.36 

.40 

.34 

50 

74 

. 26 

. 40 

.44 

,24 

.36 

100 

149 

.43 

.36 

.42 

.73 

.51 
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TABLE  8 ITEMS  RETAINED  IN  CRITERION  TEST  3 WITH 
CORRESPONDING  DIFFICULTY  LEVELS 


(Total  N 

= 1 

,232 

; 1 = 419; 

3 = 

= 194 

; 5 

= 336; 

7 = 

283) 

Skill 

Level 

Pooled 

Skill 

Level 

Pooled 

Item 

1 

3 

7 

3-5-7 

Item 

1 

3 

5 

7 

3-5-7 

1 

2 

.27 

.40 

.57 

.86 

.63 

51 

68 

.25 

.61 

.60 

.80 

.67 

2 

4 

.35 

.38 

.34 

.49 

.40 

52 

69 

.26 

.23 

.28 

.42 

.31 

3 

6 

.29 

.51 

.49 

.60 

.53 

53 

72 

.21 

.19 

.21 

.45 

.29 

4 

7 

.28 

.31 

.36 

.36 

.35 

54 

73 

.32 

.38 

.50 

.71 

.55 

5 

8 

.34 

.32 

.39 

.59 

.44 

55 

75 

.32 

.29 

.32 

.52 

.38 

6 

9 

.21 

.22 

.22 

.28 

.24 

56 

77 

.29 

.15 

.15 

.24 

.18 

7 

10 

.16 

.15 

.08 

.20 

.14 

57 

79 

.35 

.41 

.39 

.48 

.43 

8 

11 

.31 

.20 

.26 

.47 

.32 

58 

80 

.20 

.24 

.30 

.58 

.38 

9 

12 

.18 

.22 

.15 

.27 

.21 

59 

81 

.26 

.43 

.63 

.89 

.67 

10 

13 

.10 

.09 

.11 

.24 

.15 

60 

82 

.26 

.34 

.37 

.43 

.38 

11 

14 

.28 

.33 

.43 

.65 

.48 

61 

83 

.28 

.23 

.32 

.49 

.36 

12 

15 

.15 

.16 

.13 

.17 

.15 

62 

85 

.13 

.23 

.18 

.25 

.22 

13 

16 

.25 

.30 

.23 

.33 

.28 

63 

86 

.23 

.27 

.31 

.46 

.35 

14 

17 

.23 

.26 

.34 

.70 

.44 

64 

87 

.24 

.21 

.36 

.59 

.40 

15 

18 

.24 

.20 

.24 

.70 

.39 

65 

88 

.16 

.11 

.12 

.16 

.13 

16 

19 

.13 

.18 

.18 

.48 

.28 

66 

90 

.17 

.18 

.28 

.66 

.39 

17 

20 

.25 

.52 

.73 

.94 

.75 

67 

91 

.27 

.38 

.45 

.67 

.51 

18 

21 

. 24 

.27 

.37 

.59 

.42 

68 

92 

.16 

.24 

.21 

.34 

.26 

19 

22 

.38 

.27 

.24 

.46 

.32 

69 

93 

.2“ 

.20 

.31 

.69 

.41 

20 

24 

.30 

.44 

.58 

.65 

.57 

70 

101 

.25 

.19 

.26 

.72 

.40 

21 

25 

.22 

.24 

.20 

.66 

.37 

71 

102 

.53 

.43 

.49 

.57 

.50 

22 

27 

.67 

.79 

.86 

.97 

.88 

72 

103 

.20 

.19 

.27 

.55 

.3  5 

23 

28 

.30 

.48 

.48 

.61 

.52 

73 

104 

.23 

.25 

.39 

.55 

.41 

24 

29 

.37 

.53 

.71 

.89 

.73 

74 

105 

.31 

.40 

.49 

.49 

.47 

25 

31 

.17 

.19 

.16 

.44 

.26 

75 

.’37 

.17 

.25 

.22 

.30 

.26 

26 

32 

.16 

.32 

.47 

.54 

.46 

76 

108 

.41 

.46 

.57 

.65 

.57 

27 

33 

.39 

.48 

.48 

.48 

.48 

77 

109 

18 

.2A 

.25 

.40 

.30 

28 

34 

.23 

.25 

.27 

.36 

.30 

78 

113 

^30 

*28 

.43 

.71 

.49 

29 

35 

.24 

.26 

.24 

.65 

.39 

79 

115 

.27 

^26 

.26 

.50 

.34 

30 

39 

.31 

.34 

.42 

.72 

.51 

80 

117 

!i6 

!i4 

.17 

.29 

.21 

31 

40 

.16 

.11 

.12 

.41 

.22 

81 

il9 

.28 

.32 

.34 

.55 

.41 

32 

41 

.22 

.34 

.34 

.51 

.40 

82 

120 

.25 

.29 

.27 

.29 

.28 

33 

43 

.23 

.38 

.45 

.72 

.52 

83 

121 

[42 

[31 

.33 

.45 

.37 

34 

44 

.27 

.20 

.36 

.67 

.43 

84 

122 

!i9 

^24 

.39 

.78 

.49 

35 

45 

.16 

.^5 

.60 

.93 

.68 

85 

123 

45 

.50 

.64 

.82 

.67 

36 

46 

.18 

.15 

.09 

.13 

.12 

86 

126 

■32 

[36 

.55 

.7  7 

.58 

37 

47 

.25 

.15 

.14 

.45 

.25 

87 

127 

'27 

’23 

.26 

.41 

.31 

38 

49 

.19 

.17 

.24 

.35 

.26 

88 

129 

'15 

■34 

.43 

.64 

.48 

39 

51 

.50 

.57 

.67 

.81 

.69 

89 

131 

[33 

‘30 

.31 

.49 

.37 

40 

52 

.20 

.32 

.29 

.39 

.33 

90 

132 

.27 

.34 

.36 

.40 

.37 

41 

53 

23 

15 

17 

.38 

.24 

91 

135 

.32 

.35 

.42 

•47 

.42 

42 

54 

.15 

.15 

.12 

.20 

.16 

92 

136 

.66 

.76 

.81 

.92 

.84 

43 

55 

.58 

.66 

.80 

.89 

.80 

93 

137 

.13 

.33 

•23 

•30 

.28 

44 

56 

39 

.53 

.57 

.71 

.61 

94 

138 

.26 

.23 

.34 

•43 

•35 

45 

57 

.21 

.12 

.12 

.52 

.26 

95 

140 

.29 

.27 

•26 

.35 

.29 

46 

59 

.31 

.^3 

.54 

.66 

.55 

96 

143 

.52 

.60 

•66 

•75 

.68 

47 

62 

.28 

.39 

.52 

78 

.58 

97 

144 

.20 

. 12 

.12 

• 27 

.17 

48 

63 

.26 

.41 

.37 

.58 

.46 

98 

145 

. 16 

. 18 

.17 

.30 

.22 

49 

64 

[29 

]38 

.52 

.68 

.54 

99 

146 

. 38 

.29 

.40 

.60 

.44 

50 

66 

.42 

.53 

.63 

.76 

.65 

100 

149 

. 31 

.52 

.54 

.64 

.57 
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Dlscrimlnabllltv  and  Reliability  of  the 
Final  Versions  of  the  Criterion  Tests 


Following  selection  of  the  100  best  items  in  each  of  the  three 
Criterion  Tests,  the  tests  were  again  scored  using  only  the  retained 
items.  The  means  and  standard  deviations  for  each  test,  plus  the  total 
of  the  test  scores,  classified  by  skill  level,  are  presented  in  Table  9. 
The  test  mean  scores  shown  in  Table  9 now  have  a maximum  possible  value 
of  100  rather  than  150.  The  new  test  scores  are  also  corrected  for 
guessing. 


TABLE  9 MEANS  AND  STANDARD  DEVIATIONS 


OF  TOTAL  SCORES  ON  TESTS  1,  2, 
3,  AND  POOLED  TEST  SCORE  BY 
SKILL  LEVEL 


(Total  N 

= 1,232; 

1 = 419;  3 = 194; 

5 = 336; 

7 = 283) 

Test  Score 

Skill  Level 

1 

2 

3 

Total 

1 

M 

3.98 

5.34 

3.13 

12.45 

SD 

7.48 

6.56 

5.78 

13.40 

3 

M 

61.95 

15.35 

8.54 

85.84 

SD 

17.89 

11.70 

7.18 

29.56 

5 

M 

59.06 

24.00 

17.78 

100.84 

SD 

17.84 

13.10 

11.10 

35.23 

7 

M 

73.46 

45.62 

38.57 

157.65 

SD 

12.27 

13.65 

14.24 

34.07 

Pooled  3-5-7 

M 

64.76 

29.46 

21.57 

115.79 

SD 

17.36 

17.86 

17.15 

45.62 

As  was  the  case  before  final  item  selection,  the  total  scores  in 
Table  9 generally  Increase  as  a function  of  Increasing  skill  level.  This 
result  holds  over  all  three  Criterion  Tests.  One  exception  is,  again, 

CT  1 where  level  3 personnel  score  slightly  higher  than  level  5 personnel 
(61.95  versus  59.06).  A multivariate  analysis  of  variance  (MANOVA)  per- 
formed on  the  three  individual  test  scores  Indicates  that  the  group  mean 
profiles  are  significantly  different  across  skill  levels  (p<.0001). 

A corresponding  univariate  analysis  of  variance  on  the  sum  of  the  test 
scores  was  also  highly  significant  (F=15A7.71,  with  3 and  1228  DF, 
p<.0001).  The  analysis  of  variance  results  are  summarized  in  Table  10. 


TABLE  10.  LT^IVARIATE  AND  MULTIVARIATE  ANALYSIS  OF  VARIANCE  RESULTS 
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The  tests  of  the  main  effect  have  3 degrees  of  freedom,  and  the  error  terms  have  1228 
degrees  of  freedom  in  all  cases. 


The  analysis  of  variance  results  indicate  that  the  test  scores  do 
discriminate  between  personnel  at  each  of  the  skill  levels.  Correlation 
coefficients  computed  between  skill  levels  (e.g.,  1,  3,  5,  and  7)  and 
scores  on  each  of  the  three  Criterion  Tests  and  their  sum  are  shown  in 
Table  11.  The  correlations  lend  support  to  the  analysis  of  variance  re- 
sults, and  also  give  an  idea  of  the  degree  of  association  between  indi- 
vidual test  performance  and  corresponding  skill  level. 


TABLE  11  CORRELATIONS  BETWEEN  TEST  SCORES 
AND  ASSOCIATED  SKILL  LEVEL  FOR 
ALL  SUBJECTS 

(Total  N = 1,232;  1 = 419;  3 = 194;  5 = 336;  7 = 283) 


.851  .770  .723  .876 


One  of  the  primary  objectives  of  the  Criterion  Test  development  effort 
has  thus  been  achieved:  the  test  results  disciminate  very  well  among  skill 
levels.  Individual  Criterion  Test  Performance  is  also  highly  associated 
with  skill  level. 

Table  12  presents  reliability  coefficients  for  each  of  the  three  Cri- 
terion Tests  categorized  by  skill  level. 


TABLE  12  RELIABILITY  COEFFICIENTS  ON  TESTS  1,  2, 
AND  3 CLASSIFIED  BY  SKILL  LEVEL 


(Total  N = 

1,232;  1 = 419; 

3 = 194;  5 = 336; 

7 = 

Skill 

Test 

Level 

1 

2 

3 

1 

.67 

.55 

.44 

3 

.95 

.86 

.63 

5 

.95 

00 

00 

.85 

7 

.92 

.91 

.91 

Pooled  3-5-7 

.95 

.94 

. 94 

The  lallablllty  coefficients  shown  in  Table  12  are  the  KR20  version  of 
Cronbach's  coefficient  Alpha:  _ 

I I ^ '’f 

(1)  r^,.  - 1 - i=  1 
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In  Equation  1,  I is  the  total  number  of  items  in  the  test, 

2 

aj  is  the  variance  of  each  individual  item  i, 

2 

oj  = P^Cl-Pj*)  where  P^  is  the  proportion  of  respondents 
who  answer  item  i correctly  = difficulty  of  item  i. 


2 

and  Ox  *^he  variance  of  the  distribution  of  total  scores. 

Due  to  time  and  scheduling  constraints,  the  current  situation  per- 
mitted only  one  administration  of  each  of  the  Criterion  Tests.  One  test 
administration  permits  reliability  assessment  using  either  a split-half 
procedure  or  coefficient  alpha  (Kuder-Richardson  reliability).  Split-half 
reliability  estimation  was  ruled  out  because  of  the  difficulties  involved 
in  splitting  the  Criterion  Tests  into  equivalent  or  "parallel"  halves.  In 
order  to  construct  parallel  halves,  each  test  would  have  had  to  be  divided 
into  two  parts,  with  each  part  being  equivalent  in  terms  of  content,  item 
difficulty,  and  so  forth.  In  all  likelihood,  this  could  have  been  accom- 
plished; however,  a significant  increase  in  development  costs  would  have 
resulted.  Additional  costs  would  also  have  resulted  from  the  practical 
necessity  that  the  parallelism  of  the  tests  be  empirically  demonstrated. 

Coefficient  alpha,  or  f \20,  provides  information  about  the  consistency 
of  individual  performance  from  item  to  item  within  a single  test.  The 
statistic  thus  provides  an  estimate  of  the  internal  equivalence  of  the 
tests.  Coefficient  alpha  has  been  shown  to  be  a lower  bound  estimate 
of  the  true  reliability  coefficient  of  a test.  That  is,  coefficient  alpha 
will  always  be  lower  than  the  true  reliability  coefficient,  and  lower  than 
a reliability  estimate  obtained  using  another  procedure  such  as  split-half 
or  parallel  forms.  Thus,  should  coefficient  alpha  provide  an  "acceptable" 
reliability  estimate,  it  may  then  be  said  that  the  true  reliability  of 
the  test  is  also  "acceptable." 

Given  the  assumption  that  job  proficiency  is  a unidimensional  construct, 
assessing  intra-subject  response  consistency  using  coefficient  alpha  is 
acceptable.  However,  if  job  proficiency  is,  in  reality,  a complex  or  multi- 
dimensional construct,  then  stressing  internal  consistency  using  coefficient 
alpha  may  not  be  appropriate.  The  scope  of  the  present  study  does  not  in- 
clude assessing  the  factorial  nature  of  the  Criterion  Tests,  or  investigat- 
ing the  underlying  structure  of  the  concept  of  job  proficiency.  The  KR20 
reliability  coefficients  are  included  only  as  descriptive  indices  of  the 
degree  of  intra-subject  response  consistency  across  the  various  items  in 
the  tests. 

Considering  the  reliability  coefficients  presented  in  Table  12,  several 
patterns  are  apparent  which  are  worth  noting.  First,  each  of  the  tests 
provides  high  reliability/response  consistency  estimates  for  its  criterion 
group  (e.g.,  T]^  for  level  3 persornel,  etc.).  Second,  as  expected,  response 
consistency  decreases  as  the  overall  difficulty  level  of  the  tests  increases. 
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As  a final  word  on  reliability/consistency,  and  also  to  sununarize  the 
descriptive  phase  of  the  study.  Tables  13,  14,  and  15  present  item-total 
score  correlations  for  each  of  the  items  on  the  three  Criterion  Tests. 

The  correlations  are  also  presented  by  skill  level.  The  results  shown  in 
Tables  13,  14,  and  15  extend  the  information  supplied  by  the  KR20 
reliability /consistency  estimates.  That  is,  CT  1,  which  yields  the  highest 
reliability  estimates,  also  has  the  highest  and  most  consistent  item- total 
score  correlations.  This  is  particularly  true  for  level  3 and  level  5 
personnel.  Item-total  score  correlations  on  CTs  2 and  3 are  generally 
lower  and  less  consistent  than  those  of  CT  1.  This  latter  situation  is 
also  reflected  in  the  lower  reliability  estimates  for  CTs  2 and  3. 


TABLE  13  ITEM-TOTAL  SCORE  CORRELATIONS  FOR  FINAL  100  ITEMS  ON  TEST  1 


(Total  N = 1,232;  1 > 419;  3 * 194;  5 - 336;  7 * 283) 


Skill 

Level 

Pooled 

Skill 

Love  1 

I’ni)  1 I'd 

Item 

1 

3 

5 

7 

3-5-7 

Item  1 

3 

5 

7 

3-5-7 

1 

2 

.09 

.43 

.22 

.17 

.26 

51 

81 

-.00 

.20 

.19 

.18 

.21 

2 

3 

.09 

.25 

.24 

.24 

.30 

52 

82 

.18 

.22 

.41 

• 35 

.36 

3 

6 

.05 

.20 

.31 

.23 

.35 

53 

83 

.21 

.23 

.32 

.31 

.31 

4 

8 

.05 

.38 

.42 

.35 

.40 

54 

87 

.11 

.30 

.32 

.23 

.33 

5 

9 

.10 

.18 

.37 

.18 

.30 

55 

89 

.10 

.42 

.34 

.26 

.35 

6 

10  - 

-.03 

.27 

.19 

-14 

.21 

56 

90 

-.06 

.38 

.38 

.14 

.37 

7 

14 

.16 

.25 

.20 

.26 

.27 

57 

91 

. 12 

.26 

• 40 

• 04 

• 33 

8 

15 

. 20 

.27 

.24 

.27 

.34 

58 

92 

.04 

.37 

• 33 

• 36 

.40 

9 

17 

.20 

.25 

.27 

.19 

.21 

59 

93 

.04 

.22 

• 19 

• 11 

• 15 

10 

18 

. 19 

.16 

.25 

.15 

.23 

60 

94 

-.01 

.33 

• 35 

• 28 

.38 

11 

21 

.19 

.30 

.35 

.16 

.33 

61 

95 

-.07 

.48 

• 41 

• 01 

• 39 

12 

23 

.14 

. 37 

.41 

.30 

.43 

62 

96 

. 23 

.49 

• 35 

• 30 

.39 

13 

26 

.08 

.39 

.28 

.26 

.39 

63 

97 

. 18 

.38 

• 30 

.30 

• 36 

14 

27 

.08 

. 14 

.30 

.31 

.30 

64 

98 

. 13 

.28 

• 33 

.40 

.34 

15 

28 

.05 

. 25 

.28 

.15 

.19 

65 

99 

-.  05 

.43 

.32 

.37 

.39 

16 

31 

. 07 

.29 

.20 

.30 

.36 

66 

101 

. 13 

. 28 

.21 

.21 

.21 

17 

32  - 

-.03 

. 22 

.20- 

-.02 

.16 

67 

102 

.05 

. 19 

.26 

.21 

.23 

18 

34 

.20 

. 39 

41 

. 10 

.37 

68 

104 

.28 

.44 

.40 

.27 

.39 

19 

35 

.08 

.17 

.35 

. 15 

.26 

69 

105 

. 06 

.36 

.40 

.40 

.46 

20 

36  - 

-.03 

. 19 

.07 

.26 

.16 

70 

106 

. 09 

.32 

• 36 

• 31 

• 34 

21 

37 

.10 

.37 

.30 

.34 

.39 

71 

108 

. 13 

.21 

• 18 

• 18 

• 19 

22 

40 

. 13 

.18 

.40 

.23 

.30 

72 

110 

. 10 

. 36 

• 34 

.24 

• 34 

23 

41 

.05 

. 32 

. 39 

.34 

.38 

73 

111 

. 17 

. 47 

• 40 

• 36 

.41 

24 

42 

. 07 

. 21 

. 37 

. 21 

.31 

74 

112 

. 19 

. 39 

• 47 

.33 

.43 

25 

43 

. 14 

. 32 

.33 

. 25 

.32 

75 

114 

. 28 

.44 

• 43 

.37 

.45 

26 

44 

. 17 

. 29 

. 21 

. 20 

.24 

76 

115 

. 01 

. 23 

.34 

.16 

• 32 

27 

46 

.03 

.26 

. 23 

. 14 

.26 

77 

116 

. 22 

. 35 

.32 

.16 

.22 

28 

47 

. 08 

.25 

.08 

. 32 

.36 

78 

117 

-.  21 

. 23 

.36 

.13 

.28 

29 

48 

. 09 

.12 

. 30 

. 16 

. 22 

79 

119 

. 27 

. 46 

.36 

.38 

.39 

30 

49 

. 17 

. 29 

. 23- 

-.05 

. 20 

80 

120 

. 24 

. 42 

.47 

.48 

.50 

31 

52 

. 11 

.34 

.19 

. 11 

.21 

81 

121 

. 21 

. 38 

•17 

•12 

•16 

32 

54 

. 16 

. 16 

, 29 

. 35 

. 29 

82 

123 

. 25 

. 38 

•42 

•51 

•45 

33 

55 

. 08 

. 18 

. 34 

. 29 

.30 

83 

124 

. 22 

. 47 

• 49 

• 48 

•52 

34 

58 

. 15 

. 27 

. 31 

.17 

. 29 

84 

126 

. 15 

. 39 

.45 

.52 

.46 

35 

59 

.12 

. 38 

.39 

.27 

. 36 

85 

128 

-.  13 

. 41 

.37 

.21 

.40 

36 

61 

. 16 

. 28 

. 29 

. 17 

. 32 

86 

129 

. 04 

. 42 

.45 

.50 

.49 

37 

62 

. 14 

. 25 

. 30 

. 19 

. 28 

87 

131 

. 10 

. ^1 

.28 

.22 

.29 

38 

63 

. 14 

. 21 

. 23 

. 29 

. 26 

88 

132 

. 32 

. 53 

.45 

. 46 

.50 

39 

64 

. 10 

. 42 

. 38 

. 32 

. 41 

89 

134 

. 22 

. 26 

.25 

.19 

.14 

40 

65 

. 25 

. 38 

. 31 

. 26 

. 30 

90 

135 

. ^2 

. 29 

.30 

.35 

.30 

41 

66 

. 20 

. 22 

. 22 

. 10 

. 15 

91 

136 

. 20 

.A3 

.41 

.37 

.38 

42 

68 

. 16 

. 40 

. 32 

. 26 

. 35 

92 

137 

. 22 

. 32 

.37 

.41 

.44 

43 

69 

. 16 

. 43 

. 25 

. 13 

. 32 

93 

138 

! 38 

! 34 

.44 

.43 

.41 

44 

70 

. 13 

. 43 

. 24 

. 15 

. 22 

94 

140 

! 25 

."  26 

.32 

.43 

.34 

45 

71 

. 12 

. 24 

. 33 

. 13 

. 29 

95 

141 

. 24 

. 34 

• 46 

•52 

•43 

46 

72 

. 14 

. 44 

. 40 

. 32 

. 41 

96 

142 

. 24 

. 39 

•44 

• 37 

•39 

47 

74  - 

-.  01 

. 46 

. 25 

. 28 

. 30 

97 

145 

. 30 

. 34 

•29 

•50 

• 37 

48 

75 

. 08 

. 35 

. 29 

. 13 

. 24 

98 

146 

. 15 

. 29 

•30 

.47 

• 38 

49 

76 

. 03 

. 19 

. 19 

. 18 

. 15 

99 

149 

. 24 

. 30 

• 29 

• 20 

•28 

50 

77 

. 13 

. 26 

. 35 

. 31 

. 35 

100 

150 

. 38 

. 41 

•43 

• 43 

.40 

22 


TABLE  14  ITEM- TOTAL  SCORE  CORRELATIONS  FOR  FINAL  100  ITEMS  ON  TEST  2 


(Tot; 

il  N 

= ] 

.,232;  ; 

L = 419; 

3 = 194; 

5 = : 

336; 

7 = 

= 283) 

Skill 

Level 

Pooled 

Skill 

Love 

•1 

Poo  1 ed 

Item 

1 

3 

5 

7 

3-5-7 

Item  1 

3 

5 

7 

3-5-7 

1 

2 

.03 

.28 

.18 

.24 

.26 

51 

75 

. 05- 

-.10 

.14 

. 13 

.15 

2 

3 

.05 

.04 

.18 

.23 

.26 

52 

76 

. 20 

.35 

.42 

.44 

.41 

3 

4 

.04- 

.06 

.13 

.03 

.05 

53 

77 

. 07 

.11 

.07 

.09 

.15 

4 

5 

.12 

.14 

.28 

.19 

.31 

54 

78 

.08 

.16 

.30 

.50 

.47 

5 

7 

. 18 

.02- 

.03- 

.13 

-.02 

55 

80 

. 12 

.26 

.34 

.44 

.44 

6 

9 

.08 

.28 

.33 

,34 

.44 

56 

82 

. 20 

.29 

.29 

.41 

.38 

7 

12 

.11 

.20 

.21 

.18 

.22 

57 

83 

. 17 

.29 

•28 

.18 

• 23 

8 

13 

.13 

.25 

.12 

.18 

.32 

58 

84 

. 06 

. 10 

• 16 

.14 

• 22 

9 

14 

.06 

.27 

.20 

.26 

.46 

59 

85 

.07 

. 16 

.04 

.31 

• 28 

10 

16  - 

.00- 

.00 

.16 

.35 

.47 

60 

86 

. 04 

.38 

.26 

.20 

• 28 

11 

18 

.11 

.22 

.22 

.31 

.39 

61 

88 

. 24 

. 33 

•29 

•18 

•30 

12 

19 

.20 

.24 

.31 

.31 

.37 

62 

89 

. 18 

. 41 

•36 

•30 

•36 

13 

20 

.14 

.16 

.16 

.26 

.29 

63 

90 

. 24 

. 34 

■38 

•48 

•49 

14 

22 

.17 

.32 

.33 

.30 

.42 

64 

92 

. 11 

.07 

.04 

.00 

-.01 

15 

24 

.03 

.37 

.19 

.22 

.39 

65 

93 

.06 

.31 

• 22 

.31 

.42 

16 

25 

.23 

.29 

.36 

.26 

.34 

66 

99 

. 12 

.23 

.34 

.49 

.46 

17 

26 

. 11 

.31 

.07 

.25 

.28 

67 

100 

. 19 

.17 

.18 

.30 

.35 

18 

27 

.01 

.10 

.13 

.27 

.30 

68 

101 

.02- 

-.12 

.13 

.29 

.25 

19 

29 

.12 

.13 

.21 

.27 

.16 

69 

103 

.06 

.07 

.23 

.33 

.32 

20 

30 

.14 

.27 

.17 

.26 

.24 

70 

104 

. 11 

.22 

.36 

.53 

.55 

21 

31 

.06 

.26 

.23 

.22 

.36 

71 

105 

. 10 

.11 

.09 

.04 

.18 

22 

32 

.08- 

.15- 

.01- 

.01 

-.01 

72 

106 

. 15 

.22 

.15 

.04 

.12 

23 

33 

.22 

.32 

.37 

.38 

.42 

73 

107 

-.02 

. 21- 

.01 

.16 

.05 

24 

34 

.06 

.05 

.14 

.28 

.31 

74 

108 

. 08 

.14 

.12 

.18 

.18 

25 

35 

.14 

.07- 

.04 

.16 

.05 

75 

109 

. 13 

.29 

.26 

.37 

.36 

26 

37 

.20 

.32 

.36 

.27 

.34 

76 

111 

. 14 

. 30 

.19 

.27 

.28 

27 

38 

.08 

.30 

.30 

.31 

.34 

77 

112 

.20 

. 32 

.28 

.28 

.34 

28 

39 

.16 

.32 

.22 

.28 

.36 

78 

114 

. 12 

.20 

.27 

.10 

.19 

29 

42 

.08 

.19 

.24 

.19 

.27 

79 

116 

. 02 

. 22 

.37 

.44 

.36 

30 

43 

.19 

.43 

.39 

.28 

.41 

80 

117 

.15 

.12 

.16 

.29 

.25 

31 

45  - 

.02- 

.01 

.09 

.20 

.23 

81 

118 

. 09 

. 19 

.25 

.31 

.24 

32 

46  - 

.05- 

.03 

.16 

.25 

.38 

32 

119 

. 19 

.07 

.02 

.11 

.00 

33 

47 

.04 

.05 

.14 

.15 

.24 

83 

121 

. 09 

. 13 

.14 

.15 

.37 

34 

48 

.14 

.13 

.15 

.27 

.30 

84 

123 

.03 

. 24 

.35 

.52 

.56 

35 

50 

.35 

.27 

.39 

.29 

.38 

85 

124 

. 11 

. 17 

.21 

.38 

.29 

36 

51 

.15 

.34 

.20 

.11 

.28 

86 

127 

. 11 

. 09 

.04- 

.17 

-.03 

37 

52 

.11 

.15 

.08 

.15 

.19 

87 

129 

. 08 

. 21 

.22 

. 30 

.28 

38 

55  - 

.04 

.13 

.18 

.27 

.25 

88 

130 

. 08 

. 29 

.45 

.45 

.48 

39 

57 

.10 

.07 

.12 

.27 

.39 

89 

131 

. 14 

. 02 

. 27 

.42 

.41 

40 

59 

.23 

.36 

.29 

.28 

.28 

90 

133 

. 11 

. 02 

.01 

.29 

. 14 

41 

60 

. 12 

.24 

.35 

.28 

,32 

91 

134 

. 23 

. 36 

. 38 

.46 

00 

42 

62 

.08 

.35 

.31 

.24 

.44 

92 

135 

. 04 

. 26 

. 25 

. 22 

.42 

43 

64 

.10 

.17 

.20 

.28 

,28 

93 

136 

. 03 

. 15 

. 22 

. 40 

. 37 

44 

65 

.02 

.25 

.17 

,10 

,17 

94 

138 

. 03 

. 28 

. 37 

. 44 

.49 

45 

66  - 

.09 

.03- 

.05 

.10 

,14 

95 

139 

. 14 

. 06 

.01- 

•.  17 

-.  00 

46 

67 

.18 

.31 

.28 

.25 

.34 

96 

142 

. 20 

. 15 

. 16- 

-.06 

. 11 

47 

68 

.05- 

.02 

,09- 

,01 

.02 

97 

144 

. 02- 

-.  02 

. 11 

. 21 

. 13 

48 

69 

.23 

.30 

.37 

.39 

.47 

98 

145 

. 05 

. 17 

.22 

00 

.38 

49 

71 

.14- 

.12 

.19 

.19 

.24 

99 

147 

.06 

. 14 

.17 

.24 

.21 

50 

74 

.09 

.29 

.19- 

.03 

-.02 

100 

149 

. 07 

. 18 

.21 

.12 

.34 

23 


TABLE  15  ITEM-TOTAL  SCORE  CORRELATIONS  FOR  FINAL  100  ITEMS  ON  TEST  3 


(Total 

N = 

1. 

232; 

1 = 419; 

3 = 

194; 

5 = 

■ 336;  7 

= 

283) 

Skill 

Lev 

el 

Pooled 

Ski  1 1 

Li’ve 

1 

Poo  1 ed 

Item 

1 

3 

5 

2 

3-5-7 

Item 

1 

3 

‘5 

7 

3-5-7 

1 

2 

.15 

.20 

.27 

.29 

.42 

51 

68 

. 15 

.39 

.31 

.27 

.33 

2 

4 

.17 

.16 

.17 

.12 

.19 

52 

69 

.10- 

.06 

.20 

.19 

.23 

3 

6 

.22 

.19 

.22 

.31 

.24 

53 

72  - 

.01 

.05 

.25 

.28 

.35 

4 

7 - 

.01 

.11 

.18 

.14 

.12 

54 

73 

.12 

.08 

.20 

.35 

.35 

5 

8 

.16 

.09 

.16 

.30 

.30 

55 

75 

.15 

.21 

.24 

.19 

.29 

6 

9 

.09- 

.04- 

.01 

.14 

.09 

56 

77 

.07 

.07 

.09 

.05 

.13 

7 

10 

.05- 

.02- 

.03 

.03 

.09 

57 

79 

.12 

.25 

.31 

.23 

.23 

8 

11 

.11 

.16 

.26 

.22 

.33 

58 

80 

.17 

.07 

.20 

.41 

.39 

9 

12 

. 10 

.03 

.10 

.15 

.15 

59 

81 

.09 

.17 

.25 

.28 

.41 

10 

13 

.04- 

.17 

.10 

.16 

.20 

60 

82 

.10 

.22 

.24 

.19 

.19 

11 

14 

.26 

.29 

.32 

.35 

.39 

61 

83 

.12- 

.07 

.20 

.23 

.27 

12 

15 

.02- 

.02- 

.06 

.14 

.05 

62 

85 

. 13 

.13- 

.05 

.07 

.06 

13 

16 

.08 

.20 

.07- 

-.02 

.08 

63 

86 

.08 

13- 

.04 

.13 

.17 

14 

17 

.11 

.07 

.29 

.27 

.43 

64 

87 

.04 

.19 

.19 

.15 

.32 

15 

18 

.12 

.16 

.37 

.23 

.50 

65 

88 

09- 

.14 

.01 

.08 

.06 

16 

19 

.03- 

.03 

.18 

.17 

.33 

66 

90 

■ 01- 

]05 

.33 

.39 

.49 

17 

20 

.11 

.10 

.26 

.25 

.38 

67 

91 

.04 

.02 

.14 

.37 

.32 

18 

21 

.11 

.03 

.23 

.35 

.36 

68 

92 

.08 

.02 

.10 

.27 

.20 

19 

22 

.11 

.15 

.21 

.38 

.33 

69 

93 

. 06 

. 14 

•25 

.35 

.48 

20 

24 

.13 

.21 

.10 

.25 

.22 

70 

101 

. 09 

. 12 

.25 

.46 

.54 

21 

25 

.08- 

.02 

.24 

.33 

.46 

71 

102 

. 15 

. 23 

.20 

.19 

.21 

22 

27 

.18 

.34 

.31 

.21 

.32 

72 

103  - 

. 03- 

■.  07 

.18 

.22 

•34 

23 

28 

.20 

.21 

.16 

.17 

.20 

73 

104 

. 01 

. 19 

■ 30 

.17 

• 31 

24 

29 

.05 

.20 

.35 

.30 

.39 

74 

105 

. 15 

. 01 

.09 

.27 

. 14 

25 

31 

.07 

.01 

.16 

.30 

.34 

75 

107 

. 03 

. 10 

. 13 

. 24 

. 17 

26 

32 

.12 

.07 

.15 

.12 

.18 

76 

108 

. 19 

. 22 

. 27 

.33 

. 28 

27 

33 

.17 

.19 

.13 

.14 

.10 

77 

109 

. 09 

.10 

.12 

.24 

.23 

28 

34 

.11 

.12 

.14 

■ 11 

■ 16 

78 

113 

. 14 

.29 

.38 

.22 

.43 

29 

35 

.06 

.08 

.31 

.34 

.46 

79 

115 

. 09 

.10 

.20 

.36 

.34 

30 

39 

.10 

.09 

.21 

.32 

.39 

80 

117 

. 07 

.21 

.10 

.19 

.22 

31 

40 

.06- 

.05 

.20 

.33 

.39 

81 

119  - 

. 01 

.14 

.11- 

.01 

.19 

32 

41 

.18 

.07 

.23 

.31 

.28 

82 

120 

. 12 

.07- 

.03 

.24 

.07 

33 

43 

.10 

.13 

.20 

.15 

.32 

83 

121 

. 07 

.08 

.09 

.05 

.13 

34 

44 

.14 

.18 

.36 

.46 

.51 

84 

122 

. 15 

.16 

.26 

.40 

.50 

35 

45 

.05 

.21 

.31 

.27 

.45 

85 

123 

. 17 

.23 

.30 

.40 

.39 

36 

46 

.10 

.02 

.00 

.12 

.05 

86 

126 

. 03 

.27 

.25 

.21 

.37 

37 

47 

.05- 

.06 

.10 

.31 

.36 

87 

127  - 

. 00 

.16 

.11 

.28 

.25 

38 

49 

.05- 

.01 

.08 

.19 

.19 

88 

129 

. 11 

.10 

.25 

.38 

.37 

39 

51 

.21 

.14 

.13 

.25 

.25 

89 

131 

‘ 05 

.09 

.11 

.16 

.22 

40 

52  - 

.04- 

.02 

.09 

.05 

.10 

90 

132 

."  01 

.11 

.01 

.23 

.12 

41 

53 

.01- 

.11 

.11 

.33 

.31 

91 

135 

".  26 

.22 

.19 

.15 

.18 

42 

54  - 

.04 

.10 

.16 

.04 

.13 

92 

136 

. 17 

.33 

.34 

.16 

.29 

43 

55 

.17 

.34 

.35 

.37 

.36 

93 

137 

.04 

.21 

.06 

.03 

.06 

44 

56 

.13 

.18 

.20 

.23 

.25 

94 

138 

16 

.12 

.10 

.22 

.21 

45 

57 

.01- 

.00 

.24 

.53 

.54 

95 

140 

. 12 

.09 

.15 

.17 

.16 

46 

59 

.06 

.11 

.17 

.15 

.22 

96 

143 

.30 

.34 

.22 

.39 

.28 

47 

62 

.14 

• 18 

.28 

.31 

.40 

97 

144 

.05- 

.10 

.21 

.28 

.27 

48 

63 

.15 

.15 

.19 

.25 

.26 

98 

145 

.03- 

.02 

.03 

.17 

!i7 

49 

64 

.08 

.14 

.22 

.46 

.36 

99 

146 

. 05 

. 12 

.12 

.15 

.27 

50 

66 

.15 

.30 

.40 

.28 

.35 

100 

149 

. 16 

.10 

.20 

.25 

.21 

24 


VALIDITY  STUDIES 


The  Intent  of  the  present  study  was  to  develop  a series  of  paper-and- 
pencil  measures  of  job  performance  against  which  aptitude  tests  such  as 
the  ASVAB  could  be  validated.  The  validity  study  will,  thus,  primarily 
address  the  associations  between  the  ASVAB  scales  and  the  three  Criterion 
Tests.  Specifically,  validity  assessment  is  developed  in  four  phases: 

1.  The  relationship  between  the  ASVAB  and  the  Criterion 
Tests. 

2.  The  relationship  between  technical  school  grades 
(the  previous  criterion)  and  the  ASVAB. 

3.  The  relationship  between  technical  school  grades 
and  the  Criterion  Tests. 

4.  Cross-validation  studies. 

Validation  studies  of  this  type  usually  concern  the  regression  of  an 
outcome  measure  on  a test  score.  Phase  1 thus  embodies  the  main  features 
of  the  present  validation  effort.  Phases  2 and  3 are  included  to  assess 
the  relationship  of  the  previous  criterion  with  the  newly  developed  one. 

The  last  phase — cross-validation — is  included  to  get  an  estimate  of  the 
degree  of  shrinkage  in  predictability  involved  with  using  weights  derived 
from  one  sample  to  predict  job  performance  in  another  independent  group  of 
subj  ects. 

i 

The  Relationship  of  the  ASVAB  | 

and  the  Criterion  Tests  i 

I 

The  means  and  standard  deviations  of  each  of  the  skill  level  groups 
on  the  nine  ASVAB  scales  are  presented  in  Table  16.  A multivariate  analysis 
of  variance  (MANOVA)  performed  on  the  ASVAB  scores  indicates  that  the  skill 
level  groups  have  significantly  different  mean  profiles  (asymptotic  chi- 
square  = 458.0319  with  27  degrees  of  freedom,  p <.0001).  The  correspond- 
ing univariate  tests  indicate  that  two  ASVAB  subscales — Coding  Speed  and 
Automotive  Information — are  primarily  responsible  for  this  significant 
difference.  From  the  mean  scores  presented  in  Table  16,  it  is  noted  that 
the  basic  airmen  score  higher  than  the  older  mechanics  on  Coding  Speed, 
while  scoring  lower  on  Automotive  Information.  Coding  Speed  is  a clerical 
skill  ..:avily  dependent  upon  quickness  and  mental  agility.  Thus,  age 
logically  could  be  a factor  in  these  scores.  The  fact  that  the  older, 
more  skilled  mechanics  score  higher  than  basic  airmen  on  Automotive  Informa- 
tion is  also  not  surprising,  given  the  mechanical  exposure  of  these  subjects. 
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The  correlation  matrix  for  the  nine  ASVAB  scales,  pooled  over  skill 
levels,  is  presented  in  Table  17.  Most  of  the  correlations  between  ASVAB 
scales  are  in  the  low-to-moderate  range,  e.g.,  30's  to  40's.  One  exception 
to  this  general  result  involves  the  Information  Scales  (Shop,  Automotive, 
and  Electronic)  where  the  correlations  are  noticeably  higher.  However, 
given  the  experience  of  the  subjects  involved,  prior  expectation  is  that 
the  Information  Scales  will  be  highly  correlated.  The  correlation  matrix 
for  level  1 personnel  considered  separately  shows  approximately  the  same 
pattern,  except  that  the  Information  Scales  are  not  as  highly  correlated. 


TABLE  17  CORRELATIONS  OF  ASVAB  SCALES 
POOLED  OVER  SKILL  LEVELS 


(Total  N = 


1 

2 

3 

1 

1.00 

2 

0.39 

1.00 

3 

0.39 

0.50 

1.00 

4 

0.10 

0.23 

0.17 

5 

0.32 

0.36 

0.44 

6 

0.32 

0.50 

0.46 

7 

0.22 

0.49 

0.34 

8 

0.13 

0.39 

0.29 

9 

0.18 

0.49 

0.38 

813;  3 = 

194;  5 

= 336 

4 

5 

6 

1.00 

0.39 

0.43 

1.00 

0.53 

1.00 

0.61 

0.37 

0.60 

0.57 

0.26 

0.53 

0.49 

0.31 

0.53 

7 = 283) 

7 8 9 


1.00 

0.70  1.00 

0.68  0.65  1.00 


Table  18  shows  the  correlations  between  individual  ASVAB  scales  and 
the  three  Criterion  Test  scores  plus  the  total  test  score.  Individual 
correlation  results  are  also  presented  by  skill  level. 

None  of  the  ASVAB  scales  is  highly  correlated  with  any  of  the  Cri- 
terion Tests  within  any  skill  level.  The  individual  ASVAB  scales  that 
correlate  most  highly  with  Criterion  Test  performance  are  the  Informa- 
tion Scales  (Shop,  Automotive,  and  Electronic).  Arithmetic  Reasoning 
also  has  a slightly  higher  than  average  correlation  with  the  Criterion 
Tests. 


In  order  to  more  completely  assess  the  relationship  between  the  ASVAB 
scales  and  the  Criterion  Tests,  a stepwise  regression  procedure  was  used 
to  select  the  best  sets  of  ASVAB  predictors  for  the  test  scores.  Stepwise 
regression  is  used  to  achieve  a compromise  between  the  Increased  precision 
gained  by  including  as  many  predictors  as  possible,  and  the  practical 
necessity  of  reducing  complexity  and  costs  by  including  only  as  many 
variables  as  necessary.  A stepwise  regression  procedure  enters  into 
regression  only  those  variables  with  an  independent  contribution  to  pre- 
diction that  is  statistically  significant.  After  the  first  step  in  the 
process,  the  contribution  of  each  variable  already  in  regression  is  again 
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TABLE  18  CORRELATIONS  OF  ASVAB  SCALES  WITH  CRITERION 
TEST  SCORES  BY  SKILL  LEVEL 

(Total  N = 1,232;  1 = 419;  3 = 194;  5 = 336;  7 = 283) 

Skill  ASVAB  Scale 

Level  Test  123456789 


1 

.171 

.273 

.209 

.247 

.174 

.250 

.291 

.303 

.341 

2 

.081 

.216 

.240 

.218 

.196 

.178 

.255 

.225 

.334 

3 

.158 

.211 

.179 

.095 

.146 

.131 

.089 

.079 

.133 

T 

.203 

.349 

.311 

.286 

.256 

.283 

.325 

.313 

.411 

1 

.231 

.371 

.283 

.214 

.257 

.390 

.360 

.428 

.399 

2 

.010 

.231 

.201 

.149 

.264 

.275 

.114 

.281 

.275 

3 

.147 

.170 

.236 

.112 

.282 

.190 

.110 

.200 

.255 

T 

.179 

.357 

.308 

.216 

.329 

.391 

.289 

.418 

.413 

1 

.254 

.293 

.336 

.116 

.245 

.228 

.219 

.299 

.320 

2 

.063 

.159 

.234 

.064 

.111 

.134 

.169 

.211 

.255 

3 

.174 

.240 

.247 

.095 

.142 

.127 

.226 

.168 

.263 

T 

.207 

.283 

.335 

.112 

.210 

.205 

.245 

.283 

.340 

1 

.245 

.113 

.208 

.159 

.190 

.170 

.091 

.118 

.165 

2 

.197 

.175 

.229 

.147 

.193 

.200 

.178 

.216 

.235 

3 

.217 

.181 

.269 

.068 

.152 

.092 

.092 

.136 

.155 

T 

.258 

.187 

.279 

.145 

.209 

.180 

.143 

.186 

.218 

1 

.213 

.291 

.312 

.211 

.190 

.281 

.291 

.365 

.384 

2 

.024 

.189 

.235 

.210 

.052 

.204 

.275 

.362 

.379 

3 

.077 

.214 

.250 

.201 

.038 

.165 

.280 

.323 

.363 

T 

.119 

.265 

.305 

.238 

.:o7 

.249 

.323 

.402 

.431 

evaluated  to  see  if  it  is  needed,  given  that  the  new  variable  is  now  in 
regression.  The  process  usually  results  in  a regression  equation  con- 
taining those  variables  with  the  largest  individual  contribution  to 
prediction. 

Stepwise  regression  results  using  job  performance  test  scores  as  cri- 
teria and  the  ASVAB  scales  as  predictors  are  presented  in  Table  19.  The 
results  in  Table  19  are  given  separately  for  pooled  3,  5,  and  7 level 
personnel  and  for  basic  airmen  (level  1).  Basic  airmen  are  separated 
from  experienced  jet  mechanics  for  analysis  purposes  because  of  the  pos- 
sibility that  the  experience  factor  (e.g.,  time  in  service,  attendance 
at  technical  school,  job  incumbency,  and  so  forth)  would  moderate  the  re- 
lationship between  the  Criterion  Tests  and  the  ASVAB. 

Separate  regression  solutions  are  also  presented  for  the  total  sample 
and  a "computing  subsample."  The  results  involving  the  computing  subsample 
are  to  be  used  in  the  later  cross-validation  exercise.  In  all  analyses, 
the  three  most  predictive  ASVAB  scales  are  entered  into  regression. 
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TABLE  19  SUMMARY  OF  STEPWISE  REGRESSION  ANALYSES  INVOLVING 
CRITERION  TEST  SCORES  AND  ASVAB  SCALES 


Test  1 — Pooled  3, 


and  7-Level  Personnel  (Total  Sample;  N = 813) 


Variables  In  Regression — 9,  3,  and  8 
Percentage  of  Variance  Explained  (R^)  = 18.8% 
Regression  Coefficients: 

BO  = 37.07 
B9  = 0.712 
B3  = 0.525 
B8  = 0.487 

ANALYSIS  OF  VARIANCE 


Source 

DF 

!§. 

F 

P 

Total 

812 

244763.00 

Regression 

Residual 

3 

809 

45998.09 

198764.91 

15332.70 

245.69 

62.41 

.0001 

Test  1 — Pooled  3,  5,  and  7-Level  Personnel  (Computing  Subsample;  N = 402 


Variables  in  Regression — 9,  3,  and  1 
Percentage  of  Variance  Explained  = 18.0% 
Regression  Coefficients: 

BO  = 38.40 
B9  = 1.045 
B3  = 0.385 


ANALYSIS  OF  VARIANCE 


Source 

ss 

MS 

F 

P 

Total 

401 

108406. 50 

Regression 

3 

19552.29 

6517.43 

29.19 

.0001 

Residual 

398 

88854.21 

223.25 

Test  1 — Basic  Airmen  (Level-1;  N = 419) 

Variables  in  Regression — 9,  2,  and  8 
Percentage  of  Variance  Explained:  17.1% 

Regression  Coefficients: 

BO  = -11.878 

B9  = 0.338 

B2  = 0.383 

B8  = 0.219 

ANALYSIS  OF  VARIANCE 

Source  ^ ^ Z Z 

Trcal  418  23369.85 

regression  3 4003.74  1334.58  28.60  .0001 

iesidual  415  19366.11  46.67 
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TABLE  19  (continued) 


Test  2 — Pooled  3^  5,  and  7-Level  Personnel  (Total  Sample;  N = 813) 


Variables  in  Regression — 9,  8,  and  1 
Percentage  of  Variance  Explained:  17.4% 
Regression  Coefficients: 


BO  = 
B9  = 
B8  = 
B1  = - 

8.120 

0.967 

0.632 

■0.129 

ANALYSIS  OF 

VARIANCE 

Source 

M 

SS 

F 

p 

Total 

812 

259029.75 

Regression 

Residual 

3 

809 

45023.32 

214006.42 

15007.77 

264.53 

56.73 

.0001 

Test  2 — Pooled  3,  5,  and  7-Level  Personnel  (Computing  Subsample;  N = 402) 

Variables  in  Regression — 9,  5,  and  8 
Percentage  of  Variance  Explained:  18.4% 

Regression  Coefficients: 

BO  = 8.884 

B9  = 1.144 

B5  = -0.538 
B8  = 0.509 

ANALYSIS  OF  VARIANCE 


Source 

SS 

NB 

F 

P 

Total 

401 

124156.4^ 

Regression 

3 

22793.47 

7597.82 

29.83 

.0001 

Residual 

398 

101363,00 

254.68 

Test  2 — Basic  Airmen  (Level-1;  N = 419) 

Variables  in  Regression — 9,  3,  and  7 
Percentage  of  Variance  Explained:  14.1% 
Regression  Coefficients: 

BO  = -5.073 
B9  = 0.342 

B3  = 0.187 

B7  = 0.169 

ANALYSIS  OF  VARIANCE 


Source 

SS 

MS 

F 

P 

Total 

418 

17987.88 

Regression 

3 

2527.86 

842.62 

22.62 

.0001 

Residual 

415 

15460.02 

37.25 

TABLE  19  (continued) 


Test  3 — Pooled  3,  5,  and  7-Level  Personnel  (Total  Sample;  N = 813) 


Variables  in  Regression — 9,  8,  and  6 
Percentage  of  Variance  Explained:  15.3% 
Regression  Coefficients: 


BO  = 0.317 

B9  = 1.096 

B8  = 0.560 

B6  = -0.469 

ANALYSIS  OF  VARIANCE 


Source 

IS 

MS 

F 

P 

Total 

812 

238692.72 

Regression 

3 

36527.64 

12175.88 

48.72 

.0001 

Residual 

809 

202165.08 

249.90 

Test  3 — Pooled  3,  5,  and  7-Level  Personnel  (Computing  Subsample; 


N = 402) 


Variables  in  Regression — 9,  3,  and  5 
Percentage  of  Variance  Explained:  18.5% 
Regression  Coefficients: 

BO  = -0.096 
B9  = 1.240 

B3  = 0.666 

B5  = -0.608 

ANALYSIS  OF  VARIANCE 


Source 

ss 

MS 

F 

P 

Total 

401 

122331.89 

Regression 

3 

22633.00 

7544.33 

30.12 

.0001 

Residual 

398 

99698.88 

250. 50 

Test  3 — Basic  Airmen  (Level-1;  N = 419) 


Variables  in  Regression — 2,  1,  and  4 
Percentage  of  Variance  Explained:  6.8% 
Regression  Coefficients: 


BO  = ■ 

-6.724 

B2  = 

0.294 

B1  = 

0.046 

B4  = 

0.108 

ANALYSIS  OF 

VARIANCE 

Source 

DF 

SS 

IIS 

Total 

418 

13965.04 

Regression 

3 

954.08 

318.03 

Residual 

415 

13010.96 

31.35 

31 


TABLE  19  (continued) 

Total  Test  Score — Pooled  3.  5,  and  7-Level  Personnel  (Total  Sample;  N = 813) 


Variables  in  Regression — 9,  8,  and  3 
Percentage  of  Variance  Explained:  22.0% 
Regression  Coefficients: 


BO  = 

37.211 

B9  = 

2.371 

B8  = 

1.390 

B3  = 

1.073 

ANALYSIS  OF 

VARIANCE 

Source 

MS 

F 

P 

Total 

812 

1689969.00 

Regression 

3 

371977.88 

123992.63 

76.11 

.0001 

Residual 

809 

1317991.12 

1629.16 

Total  Test  Score — Pooled  3,  5,  and  7-Level  Personnel  (Computing  Subsample;  N = 402) 

Variables  in  Regression — 9,  3,  and  5 
Percentage  of  Variance  Explained:  22.6% 

Regression  Coefficients: 

BO  = 49.011 
B9  = 3.713 

B3  = 1.610 

B5  = -1.262 

ANALYSIS  OF  VARIANCE 


Source 

SS 

F 

P 

Total 

401 

815801.50 

Regression 

3 

184368.03 

61456.01 

38.74 

.0001 

Residual 

398 

631433.44 

1586.52 

Total  Test  Score — Basic  Airmen  (Level-^1 ; N = 419) 


Variables  in  Regression — 9,  2,  and  4 
Percentage  of  Variance  Explained:  24.7% 
Regression  Coefficients: 

BO  = -25.709 

B9  - 0.786 

B2  - 0.987 

B4  = 0.442 

ANALYSIS  OF  VARIANCE 
Source  DF  SS  MS 


418 

3 

415 


75095.65 

18573.60 

56522.05 


F 


P 


Total 

Regression 

Residual 


32 


6191.20 

136.20 


45.46 


.0001 


Table  19  presents  a great  deal  of  regression  information.  However, 
there  are  several  patterns  which  appear  to  cut  across  the  various  analyses. 
The  most  obvious  result  is  that  all  of  the  regression  runs  are  highly  signi- 
ficant. Next,  considering  the  analyses  involving  the  pooled  3,  5,  and  7 
level  personnel,  the  most  obvious  result  is  that  ASVAB  scale  9 — Electronic 
Information — is  entered  first  in  all  the  runs.  The  Electronics  Information 
scale  contains  questions  concerning  knowledge  of  electricity  and  electronics. 
This  scale  thus  appears  to  represent  a high  order  experiential  factor. 

ASVAB  scale  8 — Automotive  Information — is  also  Included  in  the  final  set 
of  predictors  in  all  analyses.  Automotive  Information  does  not,  however, 
have  the  consistently  high  relationship  with  Criterion  Test  performance 
of  Scale  9.  The  overall  predictive  efficiency  of  the  various  regression 
analyses  as  indexed  by  the  squared  multiple  correlation  (e.g. , the  per- 
centage of  variance  explained)  ranges  from  15.3%  for  CT  3,  up  to  22.0% 
for  the  total  test  score.  These  squared  multiple  correlations  are  com- 
parable to  those  usually  obtained  in  studies  involving  the  regression  of 
aptitude  or  intelligence  measures  on  achievement  test  performance. 

The  corresponding  regression  analyses  involving  the  basic  airmen 
indicate  a similar  pattern.  In  all  cases  except  CT  3,  Electronics  In- 
formation is  again  the  best  single  predictor  of  Criterion  Test  perform- 
ance. The  most  potent  predictor  after  Electronics  Information  is  Word 
Knowledge.  On  CT  3,  which  primarily  addresses  the  procedural  and  adminis- 
trative aspects  of  job  performance.  Coding  Speed  and  Word  Knowledge  are 
the  best  predictors  of  Criterion  Test  performance.  However,  the  relation- 
ship between  the  ASVAB  and  CT  3 for  basic  airmen  is  substantially  lower 
than  is  the  case  with  the  other  tests. 

In  summary,  the  regression  results  involving  experienced  jet  engine 
mechanics  indicate  that  the  ASVAB  information  scales,  particulary  Elec- 
tronics Information  and  Automotive  Information,  are  the  most  consistent 
predictors  of  Criterion  Test  performance.  For  basic  airmen.  Electronics 
Information  is  also  a significant  predictor;  however,  other  ASVAB  scales 
involving  academic  experience  (e.g..  Word  Knowledge  and  Arithmetic 
Reasoning)  are  more  important  than  is  the  case  with  the  experienced 
mechanics.  Taken  together,  the  regression  results  suggest  that  previous 
mechanical  or  electronic  experience  plus  general  educational  level,  as 
Indexed  by  verbal  and  arithmetic  skills,  are  the  best  predictors  of 
Criterion  Test  performance. 

The  above  results  are  not  surprising  given  the  fact  that  many  of 
the  experienced  mechanics  taking  the  tests  have  several  years  of  direct 
shop  and  mechanical  experience.  On-the-job  experiences  for  experienced 
mechanics  should  result  in  related  scores  on  the  ASVAB  information  scales 
and  the  Criterion  Tests.  Additionally,  verbal  and  arithmetic  skills  are 
usually  a factor  in  test  performance. 
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The  ASVAB  Versus  Technical 
School  Grades 


The  second  phase  of  the  validation  study  involves  the  relationship 
between  technical  school  grades  and  the  ASVAB.  Phase  2,  along  with 
Phase  3,  is  included  in  the  validation  study  for  the  following  reasons: 

1.  To  permit  a comparison  of  relationships  in  the 
present  sample  with  previously  established  relation- 
ships. 

2.  To  obtain  some  information  concerning  the  extent  to 
which  changing  the  criterion  for  selection  will 
affect  the  composition  of  the  pool  of  jet  engine 
mechanics. 

As  previously  indicated,  the  Air  Force  has  traditionally  used  tech- 
nical school  grades  (TSGs)  as  a criterion  measure  for  the  validation  of 
aptitude  tests.  Changing  the  criterion  measure  could  have  the  effect 
of  selecting  on  different  aptitude  dimensions,  thus  changing  the  com- 
position of  the  pool  of  basic  airmen  assigned  to  jet  engine  mechanic 
courses.  In  order  to  determine  whether  or  not  using  the  job  knowledge 
tests  as  criteria  will  change  the  dimensions  of  selection,  the  ASVAB 
scales  are  regressed  on  TSGs,  and  the  results  compared  with  the  previous 
finding  concerning  the  relationships  between  the  Criterion  Tests  and 
the  ASVAB. 

Table  20  presents  the  means  and  standard  deviations  of  technical 
school  grades  separated  by  skill  level.  Since  basic  airmen  have  not 
been  to  the  technical  school,  only  experienced  jet  engine  mechanics  are 
included. 


TABLE  20  MEANS  AND  STANDARD  DEVIATIONS  OF  TECHNICAL 
SCHOOL  GRADES  BY  SKILL  LEVEL 


Mean 

Standard  Deviation 
N 


Skill  Level 


3 

88.24 

5.98 

167  (86%) 


5 

86.88 

6.18 

275  (82%) 


86.83 
5.18 
89  (31%) 


The  numbers  of  observations  in  each  of  the  skill  groups  displayed  in 
Table  20  are  lower  than  the  numbers  Indicated  in  previous  analyses.  This 
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decrement  resulted  from  the  fact  that  a number  of  subjects  did  not  match 
the  Technical  Training  Course  Grade  File,  or  matched  the  file  but  did  not 
have  a course  number  of  3ABR426X2  or  3ABR432X0.  Several  subjects  matched 
the  Technical  Training  Course  Grade  File  and  attended  the  course,  but  did 
not  have  a valid  grade.  The  extent  of  subject  loss,  for  one  reason  or 
another,  is  greatest  for  the  7 level  personnel.  Only  31%  of  the  7 level 
subjects  have  valid  TSGs.  The  corresponding  capture  rates  for  level  3 
and  level  5 personnel  are  86%  and  82%,  respectively. 

The  mean  TSGs  do  not  show  much  difference  across  skill  levels. 

Mean  TSG  scores  range  from  88.24  to  86.83.  An  analysis  of  variance 
performed  on  the  TSG  mean  scores  across  skill  levels  resulted  in  an  F of 
2.76,  with  2 and  528  degrees  of  freedom;  this  value  is  not  statistically 
significant  (p  = .0625),  An  additional  observation  concerning  the  TSG 
scores  is  the  relatively  small  size  of  the  standard  deviations.  The 
range  of  standard  deviations  is  from  5.18  to  6.18.  Such  restricted 
ranges  of  TSG  scores  could  have  the  effect  of  depressing  the  relation- 
ships of  TSGs  with  the  ASVAB  scales  and  Criterion  Tests. 

The  correlations  of  the  ASVAB  scales  with  TSGs,  separated  by  skill 
level,  are  given  in  Table  21.  None  of  the  ASVAB  subscales  individually 
has  a strong  relationship  with  TSGs.  The  strongest  associations,  which 
hold  across  skill  levels,  are  ASVAB  scales  2,  3,  6,  and  9 — Word  Knowledge, 
Arithmetic  Reasoning,  Mechanical  Comprehension,  and  Electronics  Informa- 
tion. The  stepwise  regression  results  sxmmarized  in  Table  22  extend  and 
clarify  the  correlation  results. 


TABLE  21  CORRELATIONS  OF  ASVAB  SCALES  WITH  TECHNICAL 
SCHOOL  GRADES  BY  SKILL  LEVEL 


(Total  N 

= 531; 

3 = 167;  5 = 275;  7 = 

ASVAB  Scale 

89) 

Skill 

Level 

3 

1 

.129 

2 

.407 

3 

.414 

4 

.121 

5 6 

.294  .347 

7 

.237 

8 

.228 

9 

.335 

5 

.308 

.339 

.404 

.092 

.227  .277 

.224 

.283 

.333 

7 

.045 

.082 

.148 

.171 

.141  .437 

.270 

.286 

.224 

Pooled 

.227 

.321 

.369 

.109 

.240  .317 

.227 

.241 

. 298 
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TABLE  22  SUMMARY  OF  STEPWISE  REGRESSION  RESULTS 
USING  TECHNICAL  SCHOOL  GRADES  AS  THE 
CRITERION  AND  ASVAB  SCALES  AS  PREDICTORS 
BY  SKILL  LEVEL 


Scales  Included  in  Regression  Percentage  of  Variation  Explained 


Total 

Computing 

Total 

Computing 

Sample 

N 

Subsample 

N 

Sample 

N 

Subsample 

N 

Skill 

Level 

3 

3,  2,  9 

167 

3,  9,  1 

96 

28.1% 

167 

31.0% 

96 

5 

3,  9,  1 

275 

1,  3,  9 

144 

23.5% 

275 

27.2% 

144 

7 

6,  8,  2 

89 

6,  9,  3 

42 

20.5% 

89 

34.7% 

42 

Pooled 

3,  9,  1 

531 

3,  1,  6 

282 

19.6% 

531 

25. 1% 

282 

Considering  Table  22,  the  variables  entered  into  regression  are 
similar  for  the  3 and  5 level  personnel.  ASVAB  scales  3,  9,  and  either 
of  scales  1 or  2,  are  selected  in  all  cases.  For  level  7 personnel,  the 
situation  is  different.  Here,  the  regression  results  are  less  consistent, 
except  for  the  fact  that  Mechanical  Comprehension  is  always  included  in 
the  set  of  predictors. 

The  results  for  the  pooled  sample  represent  a blending  of  the  two 
previous  conditions.  In  the  pooled  sample.  Coding  Speed  and  Word  Knowl- 
edge are  included  in  both  regression  analyses.  However,  Electronics 
Information  is  included  in  the  runs  involving  the  total  sample,  while 
Mechanical  Comprehension  is  entered  into  regression  in  the  runs  involving 
the  computing  subsample. 

The  stepwise  analyses  summarized  in  Table  22  have  somewhat  higher 
levels  of  precision  than  is  the  case  for  the  regression  results  involving 
the  Criterion  Tests  and  ASVAB  scales.  In  the  regressions  involving  the 
ASVAB  and  TSGs,  the  squared  multiple  correlations  (SMCs)  range  from 
19.6%  to  34.7%.  Additionally,  the  ASVAB  scales  most  predictive  of  the 
dependent  measure  are  different  in  the  two  situations.  With  the  job- 
related  Criterion  Tests,  ASVAB  information  measures  are  the  best  pre- 
dictors. On  the  other  hand,  general  academic  skills  such  as  Word  Knowl- 
edge and  Arithmetic  Reasoning  are  most  indicative  of  technical  school 
performance.  These  latter  scales  are  also  related  to  Criterion  Test  per- 
formance, but  not  to  the  same  extent  as  with  TSGs.  It  thus  appears  that 
the  two  criteria — the  job  performance  Criterion  Tests  and  TSGs — are  re- 
lated to  slightly  different  aspects  of  individual  aptitude.  Using  the 
TSGs  as  a criterion  would  appear  to  favor  better  schooled,  more  academi- 
cally proficient  job  candidates;  using  the  Criterion  Tests  would  apparently 
favor  candidates  having  previous  experience  with,  or  interest  in, 
mechanical/electrical  tasks. 


The  Relationship  Between  the 
Criterion  Tests  and  TSGs 


In  order  to  further  clarify  the  relationship  between  the  two  criterion 
measures,  the  third  phase  of  the  validation  study  concerns  the  direct 
association  between  the  Criterion  Tests  and  technical  school  grades. 

Table  23  presents  the  correlations  between  the  three  Criterion  Tests 
scores,  their  total,  and  TSGs. 


TABLE  23  CORRELATIONS  BETWEEN  CRITERION  TEST  SCORES 


AND 

TECHNICAL  SCHOOL 

GRADES 

BY  SKILL  LEVEL 

Skill 

Level 

Criterion  Test 

N 

1 

2 

3 

Total  Score 

3 

167 

.478 

.423 

.279 

.570 

5 

275 

.608 

.381 

.244 

.525 

7 

89 

.416 

.111 

.181 

.346 

Pooled 

531 

.528 

.255 

.134 

.408 

The  correlations  presented  in  Table  23  indicate  several  results 
worthy  of  mention.  First,  the  correlations  between  TSGs  and  the  Cri- 
terion Test  scores  are  higher  for  CT  1 than  the  other  tests.  This  result 
is  not  surprising,  since  CT  1 primarily  addresses  the  job  content  of 
level  3 personnel.  The  material  taught  in  technical  school  is  most  closely 
related  to  level  3 job  skills.  The  job  content  items  in  CTs  2 and  3 do 
not  relate  as  directly  to  the  material 'presented  in  the  technical  school, 
thus  their  correlation  with  technical  school  performance  should  not  be 
as  high. 

A second  observation,  concerning  the  relationship  between  Criterion 
Test  performance  and  TSGs,  is  that,  with  one  exception,  the  correlations 
decline  with  increasing  skill  level.  This  suggests  that  job  Incumbency 
reduces  the  relationship  between  school  performance  and  the  behaviors 
measured  by  the  Criterion  Tests.  A variety  of  factors  such  as  selection 
and  on-the-job  experience  could  account  for  this  result. 

The  stepwise  regression  results  summarized  in  Table  2A  support  and 
extend  the  previous  correlation  results.  Table  24  presents  only  those 
variables  which  meet  an  entry  requirement  of  having  a partial-F  statistic 
which  is  significant  at  the  p = . 05  level.  If  no  variable  is  indicated, 
then  no  Criterion  Test  met  the  entry  requirement. 
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TABLE  24  STEPWISE  REGRESSION  RESULTS  USING 

TECHNICAL  SCHOOL  GRADES  AS  CRITERIA 
AND  CRITERION  TEST  SCORES  AS 
PREDICTORS 

(Total  N = 531;  3 = 167;  5 = 275;  7 = 89) 


Skill  Variables  In  Percentage  of  Variation 

Level  Regression  Explained 


Sample 

Total 

N 

Computing 

N 

% Total 

N 

% Computlnj 

i JL- 

3 

1,  2 

167 

1.  3 

96 

28.2 

167 

37.9 

96 

5 

1 

275 

1 

144 

36.9 

275 

31.6 

144 

7 

1 

89 

- 

42 

17.3 

89 

- 

42 

Pooled 

1.  3 

531 

1,  3 

282 

29.2 

531 

27.8 

282 

Not  surprisingly,  the  regression  results  again  indicate  that  scores 
on  CT  1 are  most  predictive  of  technical  school  performance.  CT  1 enters 
regression  first  in  all  cases.  For  skill  levels  5 and  7,  CT  1 is  the  only 
score  significantly  associated  with  technical  school  performance.  The 
squared  multiple  correlations  also  Indicate  that  the  precision  of  predic- 
tion is  higher  for  lower  skill  level  mechanics  than  for  the  more  experi- 
enced personnel. 


Cross-Validation 

The  last  phase  of  the  validation  study  involves  cross-validation  of 
the  regression  results.  In  a restricted  sense,  cross-validation  involves 
computing  regression  parameters  in  one  sample  of  subjects  and  then  assess- 
ing the  predictive  efficiency  of  these  weights  in  a similar  but  independent 
group  of  subjects. 

In  order  to  accomplish  the  above  objective,  the  total  sample  of  sub- 
jects was,  first,  randomly  divided  into  two  subsamples  denoted  the  "com- 
puting sample"  and  the  "cross-validation  sample."  Regression  parameters 
were  derived  using  the  computing  sample,  and  then  used  to  generate  pre- 
dicted scores  in  the  cross-validation  sample.  In  each  case,  weights  were 
obtained  for  the  three  most  predictive  ASVAB  scales.  Before  being  used 
to  generate  predicted  scores,  the  raw  regression  coefficients  were  con- 
verted to  Integer  weights  by  multiplying  by  10  and  then  rounding  to  the 
nearest  integer.  The  final  step  involved  computing  the  correlations 
between  actual  and  predicted  scores  using  the  subjects  in  the  cross- 
validation  sample. 

Table  25  presents  the  correlations  computed  between  the  actual  and 
predicted  scores  for  selected  situations.  Since  only  certain  Criterion 
Tests  are  relevant  for  specific  skill  groups,  e.g.,  CT  1 for  skill  level  3 
and  so  forth,  not  all  sets  of  results  are  of  interest.  Only  those 
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correlations  relevant  to  a particular  group  are  presented.  For  purposes 
of  comparison  and  evaluation,  the  squared  multiple  correlations  from  the 
corresponding  regression  analysis  involving  the  computing  subsample  are 
presented  with  the  squared  cross-validation  correlation  coefficients. 

As  is  typically  the  case  in  studies  of  this  nature,  the  squared 
correlations  computed  from  the  cross-validation  sample  show  the  shrinkage 
associated  with  applying  regression  weights  in  a second  sample.  The 
actual  magnitude  of  the  cross-validation  correlations  usually  reflects  the 
squared  multiple  correlations  obtained  in  the  computing  subsample.  How- 
ever, in  most  cases,  the  squared  correlations  are  so  low  that  they  must  be 
considered  practically  not  significant. 
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CONCLUSIONS 


Summary 

The  purpose  of  the  present  study  was  to  develop  a set  of  job-related 
Criterion  Tests  against  which  to  validate  aptitude  measures.  Toward  that 
objective,  three  150-item  job  performance  Criterion  Tests  were  developed 
and  administered  to  1232  personnel  ranging  from  basic  airmen  to  7-skill 
level  senior  jet  engine  mechanics.  By  selecting  those  items  which  best 
differentiated  skill  levels,  and  were  most  homogeneous  in  terms  of  their 
correlation  with  the  total  test  score,  the  Criterion  Tests  were  reduced  to 
a final  form  of  100  items.  Analysis  of  variance  and  correlation  results 
indicated  that  scores  on  the  Criterion  Tests  were  highly  associated  with 
actual  skill  levels.  Reliability  assessment  using  coefficient  alpha  also 
demonstrated  that  subject  responses  were  sufficiently  consistent. 

In  terms  of  the  associations  between  the  Criterion  Test  scores  and 
other  measures,  the  results  of  the  study  indicated  that  ASVAB  information 
subscales  were  most  predictive  of  Criterion  Test  performance  for  experi- 
enced mechanics.  Information  measures  were  also  related  to  test  perform- 
ance for  basic  airmen,  but  general  knowledge  subscales,  e.g..  Word  Knowl- 
edge, played  a more  prominent  role  for  these  subjects. 

When  technical  school  grades  were  regressed  on  the  ASVAB,  academi- 
cally oriented  scales,  such  as  Word  Knowledge  and  Aritlimetic  Reasoning, 
were  found  to  be  most  consistently  predictive  of  technical  school  per- 
formance. A possible  implication  of  this  result  is  that  changing  the 
standard  for  selection  as  a jet  engine  mechanic  from  TSGs  to  the  Cri- 
terion Tests  may  tend  to  select  in  favor  of  potential  candidates  with 
previous  mechanical/electrical  experience. 

As  a final  step  in  comparing  the  two  criterion  measures,  TSGs  were 
regressed  on  the  Criterion  Tests.  The  results  indicated  that  only  CT  1 
was  consistently  predictive  of  technical  school  performance.  This  result 
is  not  surprising,  since  CT  1 covers  material  similar  to  that  presented 
in  the  technical  school  course  of  study. 

The  final  phase  of  the  study  involved  cross-validation.  Here,  regres- 
sion results  from  a randomly  selected  half  of  the  subjects  were  used  to 
generate  predicted  scores  for  the  other  half.  The  derived  correlations 
between  the  actual  and  predicted  scores  demonstrated  the  shrinkage 
typically  found  in  regression  cross-validation  studies.  Given  the  low 
percentages  of  variance  accounted  for  in  the  Initial  regression  analysis, 
the  corresponding  correlations  in  the  cross-validation  phase  indicated 
a very  low  relationship  between  actual  and  predicted  scores. 
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Discussion 


Considered  as  a whole,  the  analytical  results  Indicate  a statistically 
significant  relationship  between  Criterion  Test  performance  and  the  aptitude 
measures.  The  degree  of  the  relationship  is  such,  however,  that  the  prac- 
tical significance  of  the  relationship  is  marginal.  That  is,  ASVAB  scores 
cannot  be  used  to  reliably  predict  future  job  knowledge,  as  measured  by 
the  Criterion  Tests.  The  Criterion  Test  scores  do,  however,  appear  to 
be  quite  indicative  of  the  actual  career  level  of  jet  engine  mechanics. 

The  rather  weak  relationship  between  the  Criterion  Tests  and  the  ASVAB 
found  in  the  present  study  is  typical  of  results  obtained  in  other  similar 
investigations.  When  aptitude  measures  are  used  to  predict  job  success, 
the  relationships  are  usually  attenuated  by  a number  of  "moderator"  vari- 
ables. These  moderator  variables,  such  as  age,  IQ,  geographical  background, 
ind  so  forth,  serve  to  modify  the  relationship  between  the  predictor  vari- 
able and  the  criterion  measures.  One  solution  to  this  problem  involves 
computing  separate  regression  equations  for  major  subgroups  in  the  popula- 
tion. This  could  be  done  for  low  IQ  and  high  IQ  personnel,  for  urban 
versus  rural  personnel,  and  so  forth.  Another  possibility,  which  possibly 
would  involve  less  "capitalization  on  chance,"  is  to  include  the  moderator 
variables  as  predictors  in  the  regression  equation.  This  would  be  done  by 
collecting  a broad  range  of  biographic  data  indicative  of  the  background, 
relevant  experiences,  interests,  and  personality  of  the  subjects,  in  addi- 
tion to  aptitude  information.  The  biographical  variables  are  then  used, 
separately  or  in  a combined  form,  in  an  attempt  to  account  for  variation 
that  normally  becomes  part  of  residual  error. 

The  present  Criterion  Test  development  study  Ideally  lends  itself 
to  such  a procedure.  Not  only  are  a sufficiently  large  number  of  subjects 
involved,  but  also  a considerable  amount  of  relevant  biographical  informa- 
tion has  already  been  obtained  on  the  subjects.  However,  due  to  the  con- 
tractual limits  of  the  study,  the  biographic  variables  were  not  included 
in  the  regression  analysis.  The  consideration  of  biographical  moderator 
variables  thus  remains  an  area  for  future  study. 

The  current  study  also  makes  the  assumption  that  career  progression 
is  a valid  measure  of  job  proficiency.  If  Criterion  Test  performance  is 
associated  with  career  progression,  it  is  then  assumed  that  the  Criterion 
Tests  may  also  be  considered  a measure  of  job  proficiency.  However,  in 
reality,  the  validity  of  the  first  assumption  is  open  to  criticism.  It 
might  well  be  argued  that  career  progression  is  really  a function  of 
endurance.  That  is,  personnel  who  stay  in  the  Air  Force  will  get  promoted. 
In  that  case,  actual  job  proficiency  may  only  marginally  be  a factor  in 
promotion.  In  order  to  strengthen  the  theoretical  bases  of  the  present 
study,  in  the  event  the  Criterion  Tests  are  used  to  select  personnel,  it 
would  be  necessary  to  more  completely  specify  the  relationship  between 
Criterion  Test  performance  and  actual  job  proficiency. 
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APPENDIX  1 ITEM  ANALYSIS  OF  TEST  1 FOR  3-LEVEL  AND  1-LEVEL  PERSONNEL 


R is  the  average  rank  of  (J>(0,3)  and  it (3), 


APPENDIX  1 (continued) 


is  the  average  rank  of  <t>(0,3)  and  ’^it(3). 
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is  the  average  rank  of  >;)(0,3)  and  it  (3). 
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is  the  average  rank  of  $(0,3)  and  it (3). 


APPENDIX  1 (continued) 


is  the  average  rank  of  (J)(0,3)  and  it (3). 
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is  the  average  rank  of  ()>(0,3)  and  it (3). 
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is  the  average  rank  of  !i)(0,3)  and  ^it(3). 


APPENDIX  2 ITEM  ANALYSIS  OF  TEST  2 FOR  5-LEVEL  and  3-LEVEL  PERSONNEL 


R is  the  average  rank  of  (j)(3,5)  and  it (5). 


APPENDIX  2 (continued) 


R is  the  average  rank  of  (|i(3,5)  and  "^itCS). 
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R Is  the  average  rank  of  i}i(3,5)  and  it  (5). 
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R is  the  average  rank  of  it>(3,5)  and  it (5). 
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R is  the  average  rank  of  (Ji(3,5)  and  it (5). 
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APPENDIX  2 (continued) 


R is  the  average  rank  of  ^i(3,5)  and  it (5). 


APPENDIX  3 ITEM  ANALYSIS  OF  TEST  3 FOR  7-LEVEL  AND  5-LEVEL  PERSONNEL 


is  the  average  rank  of  <ji(5,7)  and  it (7). 
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is  the  average  rank  of  <j)(5,7)  and  it (7), 
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R is  the  average  rank  of  (}i(5,7)  and  it  (7). 
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is  the  average  rank  of  <J(5,7)  and  it(7). 
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is  the  average  rank  of  ^>(5,7)  and  it  (7). 
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is  the  average  rank  of  ^(5,7)  and  it (7). 


